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ABSTRACT 

Several tests available from the Educational Testing 
Service (Princeton, New Jersey) have been used or are candidates for 
use by the Tennessee State Department of Education as initial public 
school personnel licensure endorsement area tests. The first of the 
two reports contained in this document (i.e., the Executive Report) 
describes a study of one of these tests, namely, the test of 
Educational Leadership. The study's purpose was threefold: (1) to 
determine the test's validity as a measure of knowledge, academic 
skills, and abilities required for specific initial licensure of 
principals in Tennessee; (2) to recommend a minimum qualifying score 
for applicants who would take the test if it was declared valid for 
use in Tennessee; and (3) to ascertain the relationships between the 
145 test items and a group of 15 competencies required of principals 
working in Tennessee. A panel of 13 teacher education institutional 
personnel reviewed test content; 20 local school district staff 
members assessed test item job relevance; and a separate committee of 
nine educational and lay representatives reviewed responses to make a 
decision on test validity and a recommendation of a minimum 
qualifying score. The test was judged valid for use in principal 
certification, with a specific minimum qualifying score of 500. Three 
appendixes list committee members, summarize validity data, and list 
the recommendations per test item. Following this report, the 
Technical Report presents the rationale for the following study 
parameters: (1) research design and procedures; (2) functions of the 
evaluating panel of 33 educators; (3) functions of the 9-member 
Standards Committee; (4) organization of the panels; (5) formation of 
the panels; (6) preparation for panel meetings; (7) data collection; 
(8) selection and meeting of the Standards Committee; and (9) data on 
panel responses. A concern of professional educators is to conduct 
similar periodic reviews of all tests required for licensure in 
Tennessee to ensure that recently created and revised tests are 
analyzed for job relevance. Six tables present data from the study. 
Eight appendixes describe the test's content and provide instructions 
for, overviews of, and materials concerning tasks to be performed by 
members of the Content Review Panel, the Job Relevance Panel, the 
Knowledge Estimation Panel, and the Standards Committee. (SLD) 
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Introduction 

Statutory requirements for initial licensure/certification and endorsement of public school 
personnel in Tennessee were established by the Comprehensive Education Reform Act of 1984. The 
legislatioia mandated that applicants for initial certification must present minimum qualifying 
scores on secured tests of communication skills, general knowledge, professional knowledge, and 
endorsement area specializations. The act stipulated that the requirements would become effective 
July 1, 1984, or as soon thereafter as the tests could be validated and have minimum qualifying scores 
established. 

A statewide study, conducted in 1984, determined that the three National Teacher 
Examinations (NTE) Core Battery tests and 23 of the 25 NTE Specialty Area Tests were valid to use 
as initial certification tests in Tennessee. The study also provided the data to establish minimum 
qualifying scores for the valid tests. Upon completion of the study, the Tennessee State Board of 
Education (TSBE) immediately established minimum score requirements for the NTE Core Battery 
covering communication skills, general knowledge, and professional knowledge. Subsequently, 
the TSBE instituted minimum score requirements as recommended for NTE Specialty Area tests that 
correspond to 14 endorsement areas. 

Several tests available from Educational Testing Service (ETS), Princeton, NJ, have been 
used or are candidates for use by the Tennessee State Department of Education (TSDE) as initial 
licensure endorsement area tests. The TSDE obtained the services of Memphis State University to 
conduct a study of the Educational Leadership test during the summer of 1992 to determine its 
validity, to recommend a minimum qualifying score for applicants who would take the test for 
licensure as a principal, and to establish relationships between test items and competencies required 
of principals. The statewide study involved personnel from public education agencies and higher 
education institutions with professional preparation programs in these specialized areas. 

ObiectivPfi of fhA Sf iiHy 

This study had three objectives: (1) to determine the validity of the Educational Leadership 
test as a measure of the knowledge, academic skills, and abilities required for specific initial 
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licensure of principals in Tennessee and the knowledge estimation of the same test, (2) to formulate 
a recommendation on a minimum qualifying score for the test, if it was declared to be valid for use in 
Tennessee, and (3) to ascertain the relationships between the 145 items on the test and a group of 15 
competencies deemed necessary for principals who will work in Tennessee. The study was 
delimited to potential use of the tests for initial licensure rather than to select personnel for 
employment. 

Description of thp Tftst 

The test included in the study is an instrument administered by Educational Testing Service, 
a private, non-profit testing organization: Educational Leadership. 

Strategy 

The methodology employed in the study involved a panel consisting of teacher education 
institutional personnel in the review of test content, local school district professional staff in the 
assessment of test item job relevance, and both types of personnel in the estimation of knowledge 
levels by test item among minimally qualified applicants for the licensure area. The content review, 
job relevance review, and the knowledge estimation function were applied to the test. A separate 
Committee was named to review the data analyses of the responses from the panel in order to make a 
decision on test validity and a recpmmendation on a minimum qualifying score for the test, if 
declared to be valid, as well as to compare all items on the test with the 15 competencies for principals. 

Participant Nominfltion anri Sftlftf^tinn 

The nomination of some of the potential panel members was solicited by letter. Nominations 
were requested from the chief academic administrators of the teacher education units in Tennessee 
with approved preparation programs for the applicable specializations. Principals attending the 
Tennessee Academy for School Leaders (TASL) were enlisted as participants. 

Nominees had requisite expert qualifications, represented both gender groups, included 
relevant racial groups, and represented either institutions offering specialized preparation programs 
in Tennessee or school districts. A total of 33 personnel from higher education institutions (13) and 
local school districts (20) participated in the study. 
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Panel Functions 

A current form of the ETS test was supplied by Educational Testing Service for review by the 
panel members selected for the test The panel members worked independently in conducting the 
review based on instructions given by ETS personnel, who supervised the data collection sessions. 
One-day meetings to collect the data were held in Khoxville and Nashville during July, 1992. 

Each content review panel member, who represented a higher education institution in 
Tennessee, performed three tasks. First, the panelist examined each item on the assigned test to 
judge whether or not at least 90% of the students completing the appropriate preparation program 
would have the opportunity to acquire the knowledge or academic skills to choose the correct response 
for the item. Second, the panelist made judgments about the congruence between the proportion of the 
test devoted to each topic and the emphasis on the topic in the curriculum required for professional 
preparation. Third, the panelist indicated the degree to which the test as a whole was congruent with 
the total professional preparation program. 

Each job relevance review panel member, who wa« from a local school district, reviewed each 
test item on the assigned test to make judgnients about the relevance of the knowledge or academic 
skills to competent performance as a beginning specialized practitioner in Tennessee. The 
relevance of each item was judged as "Critical," "Important," "Acceptable," "Questionable," or "Not 
Relevant." 

The knowledge estimation panel members included the individuals who also performed the 
content review and job relevance assessments for the Educational Leadership test. Ai the test item 
level, each panelist made judgments about the difficulty of each item for persons who have minimum 
levels of knowledge and academic skills necessary for competent performance as a beginning 
principal in Tennessee. 

All panel members made decisions ahont whether or not each test item related to one or more of 
the 15 competencies for which principals would be held accountable in Tennesee. A panelist could 
agiee with a choice already made by an ETS respondent, disagree with the choi^^e of the ETS 
respondent, or indicate the relationship to additional competencies by filling in a columnar bubble. 



ERIC 



3 

7 



standards Committeft 

A group of 9 educational and lay representatives was selected to serve as the Standards 
Committee for the study. The committee met in Nashville on September 17, 1992, to perform three 
tasks. First, the committee reviewed the data on appropriateness (content review and job relevance 
review) for the Educational Leadership test to make a decision about its validity for consideration for 
use in Tennessee. Secondly, the committee developed a recommendation of a minimum qualifying 
score for the valid ETS test based on a review of tho knowledge estimation data and examinee 
performance data. Thirdly, the committee examined the results of the panelists' choices of the 
relationships of all items on the test to required competencies. 

Presentation of Data 

Two types of information are summarized in this section of the report. They are demographic 
data on panel participants and the Standards Committee and results of the panel functions (content 
review, job relevance review, knowledge estimation, and competency review). 

DemogTflp hic Data 

The personnel who performed the three panel functions were described by gender and racial 

background. The three panel groups were distributed on these variables as follows: 

new Knowledge Estimation 
(N=33) 

72.7% 
27.3% 
0% 

15.1% 
84.9% 
0.0% 

The 9-member Standards Committee was a broadly representative group of educators and lay 
personnel from Tennessee. (See Appendix A for a list of committee members.) The committee was 
described on the variables of gender and racial background as follows: 





Content Review 


Job Relevance 




(N=13) 


(N=20) 


Gender 






Male 


92.3% 


60% 


Female 


7.7% 


40% 


No Response 


0.0% 


0% 


Racial Group 






Black 


7.7% 


20% 
80% 


White 


92.3% 


No Response 


0.0% 


0% 
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Gender Male 55.5% 

Female 44.5% 

Racial Group Black 33.3% 

White 67.3% 

Content Rftviow 

The content review of the ETS tests was performed at three levels: test item, test topic, and total 
test. Appendix, B presents the results of the item review for the Educational Leadership test. The data 
indicated that over 50% of the panelists who reviewed the test reported at least 95% of the items for the 
test as content appropriate. Based on the criterion of more than 70% of the panelists, the percentages of 
test questions judged content appropriate was 88%. A minimum of 81% of the questions on the test 
were perceived as content appropriate by over 80% of the panel members. 

Appendix B also contains a derived index for the test that represents the degree of difference 
between the topical emphases of the test and the specialized preparatory curriculum. The index 
ranges from 0 (close similarity) to 100 (little similarity); the index value for the Educational 
Leadership test was 13.6. 

The data summarized in Appendix B represent the comparison of the total test and the overall 
related professional preparatory curriculum. The percentages of panelists who indicated close 
parallel or some difference for the Educational Leadership test were relatively high, or 61.5%. 

Job Relfivancft RpxHaw 

Appendix B also presents the results of the job relevance ratings of test items for the 
Educational Leadership ETS test by local school district personnel. Responses of "Critical," 
•Important," or "Acceptable" were defined as indicating relevance for items in this analysis. Over 
50% of the panelists who reviewed the test indicated that at least 97% of the items were relevant to 
competent performance as a beginning specialized practitioner in Tennessee. Over 70% of the 
panelists for the test judged about 85% or more of the items as being job relevant. 
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Knowledge Estimfttion 

The responses of -^anel members in performing the knowledge estimation function for the test 
were analyzed to derive estimated raw score means for minimally qualified certification 
applicants. Using conversion factors provided by ETS, the scaled score equivalent of the raw score 
mean was computed for the test because there were enough data to establish a mean, standard 
deviation, and standard error of measurement. Results are depicted in Table 1. 

Table 1 

SUMMARY OF THE ANALYSIS ON AN ESTIMATED SCORE FOR THE 
EDUCATIONAL LEADERSHIP TEST FOR MINIMALLY 
QUALIFIED LICENSURE CANDIDATES 





Items 


Mean 


Test Name 


Total/Scored 


Raw Score Scaled Score N 


Educational Leadership 


145/143 


67.837 528.647 33 



The scaled scores for a ETS subject-matter test can vary from a low of 250 to a high of 990, a 
740-point difference between the lowest and highest scores possible. Scores cannot be compared 
directly with another subject-matter test for two reasons: first, the tests are normed independently on 
different groups of examinees; second, the standard error of measurement, an index of the precision 
of test scores, varies to a considerable degree between tests. 

Educational Leadership Test 
The analysis of the relationship of test items to principals* competencies conducted by 30 
participants is depicted in Appendix C. Data show the number of ETS choices by item and 
disagreements with them, and ttumber of additional competencies recommended by item. Some 
items have no recommendations, whereas the participants indicate that an item (such as 10) could be 
related to nine more competencies than indicated by ETS staff members. Furthermore, as many as 



25 panelists indicated the addition of one competency flegal knowledge) to the ones indicated by ETS 
personnel for item 105. 

Standards Committee Actions 
In order to act formally as a committee, the Standards Committee elected a member to serve as 
chair during its deliberations. The decisions on test validity and recommendations on minimum 
qualifying scores and related matters are reported below. 

Test Validity Decision 

The Standards Committee was presented all data collected and analyzed in performing the 
content review and job relevance review functions. (Knowledge estimation data were not released to 
the committee until the test validity decision had been made.) The committee considered 
concurrently the content review results (test item, test topic, and total test levels) and job relevance 
review results for the Educational Leadership test independently in making decisions on test 
validity. Utilizing this approach, the committee recommended that the Educational Leadership test, 
for which a decision was required, was valid to use in Tennessee as an initial certification test. 

Recommended M inimum Qualifying Score 

The Standards Committee received the results of the knowled^^e estimation function for 
minimally qualified professional practitioners based on the judgments of the panel members who 
reviewed the Educational Leadership test. The data reported for the test were the following: number 
of examinees, scaled score mean, standard deviation, standard error of measurement, and 
knowledge estimation scaled score mean. In addition, the values for scaled score means minus 1, 2, 
3, and 4 standard errors of measurement were derived (Table 2). 

After a thorough review of the examinee performance data, the committee recommended a 
specific minimum qualifying score of 500 for the Educational Leadership test. The committee further 
recommended that the score be reviewed in two years in order that Tennessee data be considered. 
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Tablto2 

NORMATIVE AND DERIVED DATA ON THE NTE SPECIALTY AREA TEST 
ON EDUCATIONAL LEADERSHIP 



Term 


Data 


Items 




X ouu/ ocoreo. 


1 A KJt A O 

i4arl4o 


Norms 




IN 


715 








1U6 


tJ»JLJ*VH., 




Knowledge Estimation 




Mean 


529 


Mean - 




1S.E.M. 


500 


2S.E.M. 


471 


3S.E.M. 


442 


4S.E.M. 


413 



Actions Conceminy the Relations hip of Tftst Items to Competencies 

After examining the data presented them, the committee concluded that there appeared 
to be some relationship between the items on the Educational Leadership test and the competencies 
deemed necessary for principal licensure in Tennessee. There were concern;:, expressed about items 
that had few references to competencies. 
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APPENDIX A 



MEMBEBS OP THE STANDARDS COMMITTEE, 
TEST VALIDATION STUDY 

Joe Cornelius, Associate Professor of Educational Administration, Tennessee State University, 
Nashville 

Pat Gammon, Mid-Cumberland Region Director, Tennessee PTA, Hendersonville 

Tom George, Professor and Associate Dean, College of Education, University of Tennessee, 
Knoxville 

Beverly Heame, French Teacher, Central-Merry High School, uackson-Madison County Schools, 
Jackson 

Charles Jenkins, President-Elect, School Board Association, Pulaski 

B. J. Naylor, Vice-President for Academic Affairs, Freed-Hardeman University, Henderson 
Relzie Payton, Instructional Supervisor-Middle Schools, Shelby County Schools, Memphis 
Dan Russell, Director of Human Resources, Johnson City Schools, Johnson City 
Bettye Triplett, Consultant, Special Projects, Metro Public Schools, Nashville 
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APPENDIX B 

SUMMARY OP VALIDITY DATA FOR TEST ON EDUCATIONAL LEADERSHIP 



Content - l\mf\* Percentage of college personnel who rated 

item cnnt^nt as appropriate for w irnculuTn 
over over over over over 

mi Ml Mi Ml as^ u 

Percentage of items 95 92 88 81 52 13 



Content - Tnpipg 



Index 



Difference in relative emphasis 
of curriculum anH test tiipii^g 
13.6 



R 
13 



Content - Tnf^] f^c^f, 



Percentage of items 



Percentage of college personnel choosing each 
response option as nppropriate for curricuhin^ ** 
Close Some Appreciable Little 
parallel differenpefi differences similarit y 
0 615 aO£ 7.7 



R 
13 



Job rftlfivanrft - TtftiriR* Percentage of public school personnel 

who rated item content as rele vant to ^oh 
over over over over over 
Ml Ml mi Ml Ml R 

Percentage of items ^ 97 92 85 74 46 20 

♦Number of items = 145 

♦♦Percentages may not equal 100% for a teat due to omissions. 
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APPENDIX C 



NUMBER OF ETS CHOICES, DIAGREEMENTS WITH ETS CHOICES, 
ADDITIONAL COMPETENCIES RECOMMENDED BY ITEM, AND 
MAXIMUM RECOMMENDATIONS OF A COMPETENCY 



Maximum 

Disagreement Additional Recommendations 
Number of with ETS Competencies of a Competency (at 

Item ETS Choices Choices Recommended least 7) 



, scored 


0 


0 


0 


2 


2 


2 


4 


3 


1 


0 


7 


4 


3 


1 


5 


5 


1 


1 


6 


6 


1 


0 


3 


7 


1 


1 


6 


8 


1 


0 


3 


9 


4 


3 


3 


10 


1 


1 


9 


11 


1 


1 


4 


12 


2 


0 


4 


13 


4 


2 


1 


14 


1 


0 


6 


15 


3 


1 


4 


16 


2 


0 


4 


17 


2 


2 


5 


18 


5 


2 


3 


Id 


2 


0 


8 


20 


2 


0 


2 


21 


3 


2 


2 


22 


1 


0 


3 


23 


1 


0 


5 


24 


2 


0 


1 


25 


1 


0 


5 


26 


5 


2 


1 


27 


2 


1 


5 


28 


1 


0 


3 


29 


1 


1 


5 


30 


3 


2 


4 


31 


4 


2 


0 


32 


1 


1 


8 


33 


5 


d 


5 


34 


2 


2 


4 


35 


1 


0 


3 


36 


4 


1 


4 


37 


1 


0 


4 


38 


3 


3 


4 


99 


3 


2 


4 


40 


3 


1 


8 


41 


1 


1 


7 


42 


2 


0 


4 
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43 


1 


1 

X 


Q 
O 


44 


1 


1 

1 


D 


45 


O 
O 


o 

A 


4 


46 


1 

X 


1 
1 


0 


47 


Q 

V 


o 


o 


48 


9 


1 


2 


49 


o 
o 


1 
1 


Z 


SO 


o 


u 




51 


o 




1 


52 


O 


1 
1 


7 


53 


3 


1 

1 


0 


54 


2 


u 


D 


55 


1 

X 


u 


7 


56 


1 

± 




0 


57 


O 


A 

u 


4 


58 


1 

X. 


A 

u 


4 


59 


1 

X. 


A 

u 


2 


60 


O 


1 
1 


4 


61 


1 

X. 


A 

u 


Q 
O 


)t scored 


n 

V 


A 

u 


0 


63 


2 


1 

X 


0 


64 


o 


A 
U 


1 


65 


2 


1 
1 


A 


66 


1 

X 


A 
U 


1 


67 


1 

X 


A 
U 




68 


4 


O 


o 


69 


2 




A 
U 


70 


1 


0 




71 


5 


q 
o 


4 


72 


1 

X 


1 


/ 


73 


A 
*x 




5 


74 


1 
X 


A 
U 


2 


75 


1 

X 


A 
U 


6 


76 


1 

1 


A 


3 


77 


O 

4» 


1 
1 


o 
O 


78 


0 

o 


A 

u 


4 


79 


1 

X 


A 
U 


Q 
O 


80 


2 


1 
± 


1 
1 


81 


1 


1 
X 


0 


82 


1 


0 


n 


83 


1 


n 

v 


O 


84 


6 


1 

± 


0 


85 


2 


0 

v 


0 


86 


2 


n 

v 




87 
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A 
U 




88 


K 




1 


89 


1 
X 


1 
1 


7 


90 


1 


1 
1 


Q 
O 


91 


2 


1 
1 


Q 
O 


92 


4 


1 


4 


93 


4 


0 


1 


94 


1 


1 


3 


96 


3 


0 


4 


96 


3 


2 


1 


97 


3 


1 


5 



7 
7 



14 
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IntTQduetian 

Statutory requirements for the use of tests in the initial licensure/certification and 
endorsement of public school personnel in Tennessee were established by the Comprehensive 
Education Beform Act of 1984. The legislation mandated that applicants for initial certification 
must present minimum qualifying scores on secured tests of communication skills, general 
knowledge, professional knowledge, and endorsement area specializations. The act stipulated 
that the requirements would become effective July 1, 1984, or as soon thereafter as the tests could be 
validated and have minimum qualifying scores established. 

A statewide study conducted in 1984 determined that the three National Teacher 
Examinations (NTE) Core Battery teste and 23 of the 25 NTE Specialty Area Tests were vaUd to 
use as initial certification tests in Tennessee. The study also provided the data to establish 
minimum qualifying scores for the valid tests. Upon completion of the study, the Tennessee State 
Board of Education (TSBE) immediately established minimum score requirements for the NTE 
Core Battery covering communication skills, general knowledge, and professional knowledge. 
Subsequently, the Board instituted minimum score requirements as recommended for NTE 
Specialty Area teste that corresponded to subject-matter endorsement areas. 

The Educational Leadership test available from Educational Testing Service (ETS), 
Princeton, New Jersey, has been used by the Tennessee State Department of Education (TSDE) as 
an initial licensure test. The Department obtained the services of Memphis State University to 
conduct a study of the test during the summer of 1992 to determine ite validity, to recommend a 
minimum qualifying score for appUcante who would take the test for licensure as a principal, and 
to estabUsh relationships between test items and competencies required of principals. 

QhiftctivAs of tht, RhiAv 

The study conducted for the Tennessee Stete Department of Education by Memphis State 
University had three objectives. They were: (1) to determine the validity of the Educational 
LeadeTBhip te«t as a meaBure of the knowl-dge, academic BkiUs, and abilities required for a 
specific licensure of public school personnel in Tennessee, (2) to formulate a recommendation on 



a minimum qualifying score for the test if it is declared valid for use in Tennessee, and (3) to 
establish relationships between test items and competencies required of principals. The study was 
delimited to potential use of the test for initial licensure rather than tn select personnel for 
employment. 

DftagriptionofthftTesi 

The Educational Leadership test included in data collection in 1992 was constructed by 
Educational Testing Service, which is a private, non-profit testing organization. A content 
description of the test is presented in Appendix A 

MftthftdQlogv 

The design and procedures used in the study represent essentially a replication of 
comparable studies of ETS/NTE tests conducted by ETS (Faggen, 1983) and Memphis State 
University (Bowman, Bowyer, Petry, Rakow, Nothem, & Jacobs, 1984; Bowman, Petry, Emanuel, 
& Bellott, 1988; Bowman, Petry, Rakow, Douze!>:s, & Emanuel, 1989; Bowman, Petry, Rakow, & 
Emanuel, 1990; Bowman, Petry, Rakow, & Emanuel, 1991; and Bowman, Petry, Rakow, & Watt, 
1992). This section of the report presents the rationale for the design and procedures, the panel 
(unctions, the Standards Committee functions, organization of the panels, formation of the panels, 
preparation for panel meetings, the data collection, selection and meeting of the Standards 
Committee, and the data on panel responses. 

Rfltionalft for thp Mftthodology 

The literature on standard-setting for licensure examinations does not demonstrate a 
singular appropriate or clearly superior methodology. The use of professional judgment under 
relatively controlled conditions as employed in this study complies with accepted measurement 
principles and has been widely utilized in similar studies. The procedures used in making 
judgments deal with standard setting (judgments about levels of test performance necessary for 
persons who are minimally qualified in specialized areas). Consequently, the methods used in 
the study required (1) a review of the test content to determine congruence with professional 
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preparation programs, (2) a review of the test content for relevance to the job performed by initially 
licensed professional personnel, and (3) the estimation of the minimum qualifying score for 
licensure. 

The comparison of the ETS test with the professional preparation programs on a statewide 
basis required an evaluation of the specific questions and an assessment of the sections or topics 
within the test. The determination of job relevance of the test entailed an evaluation of each 
question on the test. The estimation of performance levels expected of minimally qualified 
appKcants for Hcensure necessitated a broad-based familiarity with the responsibilities of 
professional public education personnel in Tennessee. These requirements were addressed by 
systematically collecting and analyzing the judgments of representative groups of professional 
educators with appropriate experiences in local school districts and higher education institutions. 
(The nomination and selection procedures for participants are described below.) 

The study participants were assigned specific functions to be performed: content review, 
job relevance review, and knowledge estimation. Personnel who were knowledgeable with 
respect to preparation of professional public school specialists in the specialized area conducted the 
content review. Practitioners in the specialized area performed the job relevance review. Both 
types of personnel participated in the knowledge estimation activities. An independent committee 
utilized the results fi^om the test review and knowledge estimation to make a decision on test 
validity and a recommendation on a minimum qualifying score. 

Human judgment by professional educators was unquestionably the foundation of the 
study. While judgments by professionals are an integral part of all research, such judgments 
were used to generat«» the raw data that were analyzed sUtistically in this study. This strategy for 
daU collection was necessary to provide information needed by the Tennessee State Department of 
Education to determine the minimum qualifying score of the test in the initial certification and 
endorsement process. The alternative strategies available and described in the literature were 
inadequate because they do not encompass all functions required for standard setting. 
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The validation ai^pect of the study complies with the professional guidelines promulgated 
in Standardg for Edncfttinnfll and Psychological Tests (American Psychological Association, 
1974). The standards for content validation are the following: 

Evidence of content validity is required when the test user wishes to estimate how 
an individual performs in the universe of situations the test is intended to represent 
.... To demonstrate the content validity of a set of test scores, one must show that 
the behaviors demonstrated in testing constitute a representative sample of 
behaviors to be exhibited in a desired performance domain, (p. 28) 
Consequently, the content review by higher education faculty members focused on the congruence 
between the content domains of the test and the related domains of the professional preparation 
programs in Tennessee. 

The "General Standards for Validity Studies" in the TTniform Gmdelines on F^mplovee 
Selection Prnrftdnrftg (Equal Employment Opportunity Commission, 1978) states that: 

Evidence of the validity of a test or other selection procedure throuigh a construct 
validity study should consist of data showing that the procedure measures the 
degree to which candidates have identifiable characteristics which have been 
determined to be important in successful performance in the job for which the 
candidates are to be evaluated, (p. 38298) 
Therefore, job relevance of the test was included as an additional dimension of the validity 
assessment of the tests. Personnel from local school districts reviewed the test to provide 
information on the relevance of the test to the responsibilities of licenced principals in schools in 
Tennessee. 

In establishing minimum score requirements, the Standards for Edu cational and 
Psycholo gical Tftat« states, **If specific cutting scores are to be used as a basis for decisions, a test 
user should have a rationale, justification, or explanation of the cutting score adopted" (APA, p. 
66). The rationale for the recommendation on a minimum qualifying score derived in this study 
i£ that the score must be based on the collective judgments of representative experts. The personnel 
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who performed the knowledge estimation function were qualified experts from local school 
districts and professional preparatory higher education institutions throughout Tennessee. Their 
combined judgments constituted the basis for determining the minimum score on knowledge and 
academic skills required for initial licensure of professional public education personnel in 
Tennessee. 

The strategy used in the study required that the judgments of individuals from local school 
districts and professional preparatory higher education institutions would be combined on a 
statewide basis to determine the estimated performance level of minimally qualified personnel 
for the test. As a consequence, two factors may produce differential effects for the graduates of 
different institutions: (1) the use of data for the state as a whole and (2) the use of an approved 
program procedure to offer specialized preparation programs at specific higher education 
institutions. Variability in the course content and grading practices are likely across institutions 
offering approved programs in the same specialization. Therefore, graduates of different 
institutions may be unequally affected by statewide standards. 

Panel Fiinrtinnft 

The functions performed in the review of the test were conceptualized as panel activities. 
Separate panels were utilized to perform content review, job relevance, and knowledge estimation 
functions for the Educational Leadership test. Panelists were asked to relate test items (145) to 15 
competencies identified by a task force as essential for all principals in Tennessee schools. The 
activities of the panels are described below. 

Contitnt Rpvifew Pani>l The members of the Content Review Panel for the Educational 
Leadership test independently performed three tasks in reviewing test content. (Instructions for 
the tasks are presented in Appendix B - Instructions and Overview of Tasks to be Performed by 
Members of the Content Review Panel.) These tasks involved a review of the test at three levels: 
question, topic, and total test Standardized oral instructions were given at the meeting to assure 
use of the appropriate reference group in performing the tasks. 



First, the panelist examined each test question on the assigned test to judge whether or not 
90% of the students completing the related preparation program would have had the opportunity to 
acquire the knowledge or academic skills needed to answer the question correctly. Instructions 
were given to base judgments on experience with students and knowledge of the program at the 
panelist*s institution. Using test questions with the correct answers noted, the panelist in effect 
classified each question as either 'Tes," "Probably Yes," "Probably No," or "No" using the 90% 
criterion. 

Second, the panelist made judgments about the congruence between the proportion of the test 
devoted to each topic and the emphasis on the topic in the curriculum required for professional 
preparation. A description of the test content was given to the panel member that listed the migor 
topics and the approximate percentages of the test questions related to each topic. The comparison 
of the topical emphases was made by indicating whether the emphasis given to each topic in the 
preparation program was the same as, greater than, or less than the emphasis given to the topic on 
the test The terms "Greater Tlian" and "Less Than" were defined as perceived differerces of 5% 
or between the preparation program and the test 

Third, the panelist made a judgment about the similarity between the total test content and 
the composition of the curriculum for the professional preparation program. The comparability as 
perceived by the panelist was expressed by choosing from four response options: "Close Parallel," 
"Some Differences, "Appreciable Differences," and "Little Similarity.'' These judgments 
provided an indication of the congruence between the total test content and the professional 
preparation of specialists* 

The order of task performance was deliberately sequenced as described above. Because the 
questions were grouped into content categories, the panelists should be better prepared to compare 
test content and curriculum content after reviewing the test questions. Therefore, tk^ tasks were 
performed in sequence at the question, topic, and total test levels. 

Job Rela vanne Pahi^I. The members of the Job Relevance Panel for the Educational 
Leadership test reviewed each question on the appropriate test to make judgments about the degree 
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to which the knowledge or academic skills related to competent performance as a licensed 
principal of a school in Tennessee. (See Appendix C • Instructions and Overview of Tasks to be 
Performed by Members of the Job Relevance Panel.) Standardized instructions were given at the 
meeting to assure use of the appropriate reference group in making judgments. The relevance of 
each test question was judged as "Critical," "Important," "Acceptable," "Questionable," or "Not 
Relevant." Each panel member made independent judgments on the relevancy of the test 
questions on the assigned test. 

KnnwlftHpft Estimntion Pnnpl The members of the Knowledge Estimation Panel for each 
test performed a single task in making judgments about performance levels of minimally 
qualified personnel who are to serve as licensed principals public educators in Tennessee schools. 
(See Appendix D - Instructions and Overview of Tasks to Be Performed by Members of the 
Knowledge Estimation Panel.) The steps involved in preparing for the task and making the 
judgments are described below. 

Each panelist was required to conceptualize a hypothetical reference group of only those 
college graduates who are likely to pursue careers as principals in the elementary or secondary 
schools, excluding persons who are likely to pursue advanced study or non-teaching careers. 
Next, the panel member was instructed to consider only those gp'aduates who are deemed 
minimally qualified as licensed principals in Tennessee (i.e., those who have the minimum 
knowledgvi and academic skills for competent performance). Because educators are thoroughly 
experienced in using this concept, they are likely to perform well and conscientiously in defining 
minimal competence because of the consequences of making incorrect judgments with regard to 
individuals. 

Each panel member was directed to make estimates about knowledge as opposed to correct 
responses that would include guessing correctly. An exercise was used at the meeting to assist the 
panelists in forming the appropriate reference group and making judgments about knowledge 
levels. In order to prepare for the panel meeting, the panelist was encouraged to obtain 
information locally about the relevant curriculum and professional preparation program. 
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Judgments were made on each test question by the panel member using a 9-point 
numerical response scale (10%, 20%, 30%, 40%, 50%, 60%, 70%, 80%, and 90%). The score points 
were not described verbally for two reasons: (1) to avoid influencing the judgments made and (2) 
to utilize the familiar and widely-used numerical basis for judging test performance. 

Tftst Tt^m Anftlvsia . Each ot the 33 panelists was asked to relate each test item to one or 
more competencies identified by a task force as necessary for principals in Tennessee schools. 
ETS personnel prepared the answer sheet (See Appendix D) indicating their choices. Panelists 
were to indicate either their agreement, by leaving the ETS choice as it was, or their disagreement, 
by either marking the ETS choice with a slash or by adding their choice by filling in a bubble. 

Standards CcmTnittee Functions 

The Standards Committee was responsible for performing three functions: (1) to decide 
whether or not the Educational Leadership test was valid for use in the licensure of professional 
public education personnel in Tennessee; (2) to develop a recommendation on a minimum 
qualifying score for the test, if declared valid, and if there were enough tests administered to 
enable norms (mean, standard deviation, and standard error of measurement) to be set; and (3) to 
establish relationships between test items and competencies required for licensure. 

Validil^y. In order to make a decision on validity, the Standards Committee reviewed data 
on the appropriateness of the Educational Leadership test for use in Tennessee. Data fi-om the 
Content Review Panel had been previously analyzed and prepared in tabular form for the 
committee. These data included the responses on the appropriateness of the test questions, the 
congruence between the proportion of each test devoted to each topic, and the emphasis on the topic in 
the professional preparation curriculum. Data from the Job Relevance Panel were also provided to 
the committee. The analyses of these data indicated the degree to which the knowledge or 
academic skills represented by the test questions were related to competent performance as a 
principal in Tennessee. 



The StaiMkrds Committee reviewed the data from the Content Review Panel and the Job 
Relevance Panel for the Educational Leadership test. The data for only these two panels were 
presented because they provided the information required to determine the validity of the test for 
use in Tennessee. Basing their actions on a review of these data, the committee made decisions 
on the validity of the test. 

Minimum Qualifying Score. The first step in developing a recommendation on a 
minimum qualifying score was the presentation to the Standards Committee of the data that were 
derived from the Knowledge Estimation Panel for the Educational Leadership test These data 
consisted of estimated scores of persons who, based on the judgments of the panelists, are 
minimally qualified for licensure as principals in Tennessee. 

The committee also received summative data derived from the results of administering 
the tests to examinees. The database for the ETS test was sufficiently large to provide normative 
information that included the standard error of measurement (a quantitative measure of test score 
precision). 

Relationships hfthwftPn Items and nomnfitftnrifig. The committee reviewed data on the 
agreement/disagreement of panelists with decisions of ETS personnel who linked test items with 
competencies deemed necessary' for principals. 

Qrgani^flfinn of the Pflnp1« 

In this study, a panel was defined as a group of experts assigned to perform a specific 
function for the test Because the panel members made independent judgments on the test, the 
panels did not actually convene as groups to perform their respective functions. 

Since the Educational Leadership test included in the study was designed for a specialized 
professional area, the number of panelists specified for the Content Review Panel, the Job 
Relevance Panel, and the Knowledge Estimation Panel was the same across all tests. The desired 
size of each panel for the Educational Leadership test was 20, equally divided into K-12 personnel 
and higher education personnel. 
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Formfltion of the Panels 

The nomination and selection of panelists utilized procedures that gave attention to expert 
qualifications, representation of gender groups, inclusion of significant racial/ethnic group 
members, and diversity of higher education institutions and local school districts in Tennessee. 
The nomination of panel members from higher education institutions was initiated by a letter of 
request from the State Department of Education, which is included in Appendix E along with 
instructions for nominating panel members and nomination forms that are traditionally sent to 
the teacher education units with preparation programs for a specialized area. Personnel from 
local school districts in Tennessee attending the Tennessee Academy for School Teachers 
participated in the project. 

The instructions for nominating panel members provided background information on the 
study. The criteria for panel member eligibility stated in the instructions were the following: 

(1) currently serving in the public schools in the area for which nominated or 
currently serving as a faculty member in a college or university offering one or 
more approved teacher education programs 

(2) a minimum of two years of experience as defined above 

(3) licensure in the appropriate area for public school nominees 

The eligibility of nominees to serve on the panels for which they were nominated was 
ascertained based on congruence between the stated criteria and the nomination form data. The 
process of selecting panelists ensured that the factors identified above were addressed to the 
maximum extent possible. A listing of the study participants who consented to be listed is 
presented in Appendix F. The letter of notification and other information sent to the panelists are 
contained in Appendix G. 
Preparation for Panel Meetingg 

In advance of the panel meetings, the panelists were sent preliminary materials related to 
their tasks (Appendices B-D). The materials for each function (content review, job relevance, and 
knowledge estimation) included an overview of the task, the instructions for performing the task. 
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a copy of the appropriate response form, and a description of the test content For the Content 
BevieAT Panel members, suggestions were included on preparation that the panelists mii^t make, 
such as reviewing course descriptions in college catalogs, consulting with subject matter 
specialists, and discussing course content with colleagues. Many panel members did not have to 
make such preparation because they were adequately prepared by their own experience. 

Knowledge Estimation j^^anel members participated in an orientation exercise that was 
designed to provide training in the maintenance of consistent standards when making 
judgments about questions with different levels of difficulty. Sample questions from ETS tests 
were presented with two estimated percentages of examinees who would know the correct answer. 
The actual percentages were used for a national sample of ETS examinees representing the second 
and fourth quintiles on the total test score distribution. These two quintiles were used to avoid 
influencing the panelists unduly when they made actual estimates later. Between the presentation 
on the questions and the percentages, the panelists made estimates of the percentage of examinees 
in each quintile who would know the correct answer. The panel members also made estimates of 
the percentage of minimally qu£»!fied examinees who would know the correct answer to each 
sample question. 

One edition of the ETS test was used in the study. Because an ETS test is secured, the 
preparation of the test materials was performed by ETS personnel. Within the test, the items were 
grouped by content topic in order to facilitate the work of the panelists. The correct answer for each 
test question was marked on the test reviewed by each panel. The number of test questions 
reviewed was 145 for the Educational Leadership test. 

Data riftllectiQn 

For the Educational Leadership test, the morning was devoted to a meeting of the Content 
Review Panel and the Job Relevance Panel; the afternoon, of the Knowledge Estimation Panel. A 
general orientation was conducted on the background of the study and the tasks to be performed by 
each panel. AfU the introductory activities were completed, the paneHsts completed the assigned 
tasks. 
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The data collection was conducted at successive meetings in Knoxville and Nashville, in 
July, 1992. Personnel from Memphis State University handled matters related to travel by the 
panelists from the institutions of higher education. ETS personnel who work with their testing 
programs and have participated previously in similar studies conducted the data collection 
sessions assisted by ^ttfiT from Memphis State University. A general orientation was conducted 
on the background of the study and the tasks to be performed by all partidpanta. 

Selection and Meeting nf the Standarda nommittPft 

A group of 9 educational and lay representatives was enlisted by Memphis State University 
personnel to serve as the Standards Committee for the study. (See ^pendix H for a list of 
committee members.) In determining the composition of the committee, attention was given to 
several factors: geographic distribution, representation of males and females, representation of 
racial/ethnic groups, and representation of higher education institutions, local school districts, 
and the general public. 

The Standards Committee met in Nashville in September, 1992. The committee reviewed 
data regarding the appropriateness (content review and job relevance) of the Educational 
Leadership test and made a decision about the validity of the test for use in Tennessee. The 
committee developed a recommendation on a minimum qualifying score for the test based on a 
review of the data available on examinee performance. It also made a recommendation about the 
relationship of the items to competencies. Personnel involved in conducting the study served as 
staff and resource persons for the committee. Personnel from the Tennessee State Department of 
Education and the Tennessee State Board of Education were observers. 

Data on Panel RAKponaP^ 

Data analyses were performed for the responses by the Content Review, Job Relevance, and 
Knowledge Estimation Panels. The procedures utilized with each set of data are discussed below. 

Content T^viaw Three aspects of the relationship between the Educational Leadership test 
and the content of professional education preparation programs in Tennessee were assessed by 
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personnel from higher education institutions in Tennessee. Firsts the members of the Content 
Review Panel from higher education institutions examined jach test question to judge whether or 
not at least 90% of the graduates of the appropriate preparation program would have the opportunity 
to acquire the knowledge or academic skills to answer the question correctly. The paneUst was 
encouraged to choose a "Yes," "Probably Yes," "Probably No," or "No" response if any basis for 
judgment existed. The analysis of the response consisted of calculating initially for each 
question the percentage of 'Tes" and "Probably Yes" responses. Then, the data were summarized 
across all questions by calculating the percentage of questions on the test that received 'Tes" and 
"Probably Yes" responses by more than specified percentages of the panelists. Data are reported 
for over 50%, over 60%, over 70%, over 80%, and over 90% of the panelists assigned to the test 

Second, the panelists compared the relative emphases placed on the major content topics in 
the Educational Leadership test to the relative emphases placed on these topics within the related 
professional preparation program. Disregarding differences of 5% or less, the relative emphasis 
of each topic in the curriculum was rated the "Same As," "More Than," or "Less Than" the 
emphasi i in the test. The analysis of the responses for the test consisted of computing the 
Difference in Relative Emphasis (DRE) Index. The number of "More Than" responses was 
subtracted from the number of "Less Than" responses for each topic and divided by the total 
number of paneUsts who rated the topic. This procedure is based on the rationale that a "More 
Than" judgment represents the same difference as a "Less Than" judgment, thereby offsetting 
each other. The DRE Index for a test was obtained by weighing the panel judgments about each 
topic (quantity computed above) by the percentage of test questions on the topic and adding the 
products without regard to their algebraic signs. The derived DRE Index represents the degree of 
difference between the topical emphases of the test and the college curriculum based on a scale 
from 0 (very close similarity) to 100 (very little similarity). The DRE Index that was calculated 
for the test is presented in the report 

Third, the Content Review Panel members for the Educational Leadership test made 
comparisons of the similarity between the test as a whole and the appropriate professional 
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preparation program. The response options for the comparison were 'Tery Close Parallel," 
"Some Differences/' "Appreciable Differences," and "Little Similarity" between the test and the 
preparation program. The nxmiber of reviewers and the percentage distribution of responses were 
calcxilated and presented for the test 

Job Relevance The assessment of the job relevance of the Educational Leadership test 
focused on the relationship between the content of the test and the job requirements for newly 
licensed principals in Tennessee as judged by local school district personnel. The five response 
options were "Critical," "Important," "Acceptable," "Questionable," and "Not Relevant." The 
initial step in the analysis for the test was to calculate the frequen<7 and percentage distributions 
of responses for each question by the panelists. Then, the percentage of respondents who rated each 
question as relevant (i.e., "Critical," "Important," or "Acceptable'^ was computed. Finally, the 
percentages of questions that were judged relevant by more than specified percentages of the 
panelists were calculated. The percentages of questions are reported for over 50%, over 60%, over 
70%, over 80%, and over 90% of the panel members assigned to the test. 

Knowlftdye Estimfltiftn. Data analyses were performed for the responses by the Knowledge 
Estimation Panel for the Educational Leadership test. The knowledge estimation function 
focused on the percentage of minimally qualified applicants for principal licensure in Tennessee 
who would know the answer to each question. Specified response choices were 10%, 20%, 30%, 40%, 
50%, 60%, 70%, 80%, and 90%. The analysis was made by calculating the mean percentage for 
each question based on the percentages chosen by the panelists. (Omissions and "Do Not Know'* 
responses were disregarded.) The mean percentage for each question was a4justed upward by 
adding the percentage of respondents who would not know the answer but would guess correctly. 
When divided by 100, the sum of the a4iusted mean percentages for all questions on a test was 
derived to provide an estimated raw score mean for minimally qualified applicants. 

Because an ETS test is a normative instrument, a scaled score must be derived from the 
raw score. A linear equation consisting of two numerical values for the test is used to compute the 

u 3^ 



scaled score for each test The scaled score is derived by adding a specified numerical value to the 
product of the raw score and a conversion factor for the score. The conversion parameters 
provided by ETS were used to obtain the scaled score mean for the test. For the test, the report 
presents the a4justed raw score mean and the scaled score mean derived from the data collected in 
the study. 

Teat ItfiTn Analyaia. Relationships between items on the Educational Leadership test and 
the 15 competencies deemed necessary for principals of schools in Tennessee were established by 
the participants. Competencies labeled were goals/expectations, research/curriculum, growth/ 
development, decision making, needs assessments, organize/implement, technology, parent 
involvement, professional development, fiscal responsibility, communicate/motivate, cultural 
values, legal knowledge, policy/politics, and communication/general public. Inasmuch as ETS 
personnel had indicated their choice(s) for each item, which were marked on an answer sheet, 
panelists -sere asked to leave each filled-in bubble if they agreed with the choice of ETS personnel 
or mark through the filled-in bubble as an indication of disagreement. If panelists thought that 
items related to other nonindicated competencies, they were to fill in the bubble in the competency 
column. 

Preaentatinn nf Data and RpftnUft 
In this section, the data are summarized for the personnel involved as are the responses of 
the panels. The results of the deKberations by the Standards Committee are also presented. 

Data nn Pergonnftl 

Background data on the 1992 panel members are reported in Table 1 for the 33 participants. 
With respect to gender, there were more males on the Content Review Panel (12) and on the Job 
Relevance Pai^el (12); 73% (24) of the Knowledge Estimation Panel were males. Concerning 
ethnicity, there were more whites than blacks on all panels; on the Knowledge Estimation Panel, 
the ratio of blacks to whites was 1 to 5.6. Demographic data on the 9 Standards Committee members 
are also presented in Table 1. With regard to gender, 56% (5) of the committee members were 
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males; when classified by ethnicity, 67% (6) of the committee members (9) were white and three 



were black. 



Table 1 

FREQUENCY AND PERCENTAGE DISTRIBUTIONS OF PANELISTS 
AND STANDARDS COMMITTEE BY GENDER AND ETHNICITY 



Gender 



Nf» 

Male Female response Total 

Function f % f % f % 



Panel 



Content Review 


12 


92 


1 


8 


0 


0 


13 


Job Relevance 


12 


60 


8 


40 


0 


0 


20 


Knowledge Estimation'* 


24 


73 


9 


27 


0 


0 


33 


Standards Committee 


5 


56 


4 


44 


0 


0 


9 



Ethnicity 



Black White Other Total 

Function f % f % f % 



Panel 



Content Review 


1 


8 


12 


S2 


0 


0 


13 


Job Relevance 


4 


20 


16 


80 


0 


0 


20 


Knowledge Estimation'* 


5 


15 


28 


85 


0 


0 


33 


Standards Committee 


3 


33 


6 


67 


0 


0 


90 
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Table 2 shows the number of 1992 participants in the test validation process from colleges 
and universities and school districts for the test. A minimum of 20 participants for each panel was 
sought. Absences and cancellations caused the numbers to be fewer for the colleges and 
universities; however, each institution with a preparation program for principals was 
represented. Adequate numbers participated in each category, which allowed the statistical 
analyses to proceed. 

Table 2 

NUMBER OF PARTICIPANTS FROM COLLEGES/UNIVERSITIES 
AND SCHOOL DISTRICTS FOR THE EDUCATIONAL 
LEADERSHIP TEST 



Test Colleges/ School Total Number 
Universities Districifl nfPflTfi<Tipftnf.Q 

Educational Leadership 13 20 33 



Table 3 contains data on the number of participants for panel functions for the test. The 
total number of participants for the content review function (colleges/university personnel) was 
13. The job relevance participants (school district personnel) totaled 20. The knowledge 
estimation group (both college/university and school district personnel) totaled 33. 



Tabled 

PANEL FUNCTION PARTICIPANTS FOR THE EDUCATIONAL LEADERSHIP TEST 







Panel Function 




Test 


Content 


Job 


Knowledge 




Review 


Relevance 


Estzmation 






(N) 




Educational Leadership 


13 


20 


33 
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Content I^view Panel Results 

The Content Review Panel for the Educational Leadership test conducted reviews of the 
questions and topics of the test The results of the analyses at each level are reported below. 



Cuntent Anpropriatiinegg of Mnltinle-ClhQic e Test Questionft . The Content Review Panel 
members examined each question on the assigned test to judge whether or not 90% of the students 
completing the appropriate college program would have the opportunity to acquire the knowledge or 
academic skills to answer the question correctly. After calculating the percentage of *Tes'* and 
"Probably Yes" responses for each question^ the percentage of items receiving more than specified 
per< entages of **Yes" responses was computed to obtain a measure of content appropriateness. The 
results are reported in TiJ>le 4. 

Table 4 

SUMMARY OF VAUDITY DATA ON THE EDUCATIONAL LEADERSHIP TEST 



Content - Tfi>ni«* 

Percentage of items 
Content - TopirR 



Percentage of college personnel who rated 
item content as appropriate for curriculum 
over over over over over 
StSSi flS^ flOSb 90% 

96 92 88 81 52 
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Index 



Difference in relative emphasis 
of curriculum and test tnpigs 



13 



Content -Tftf^lW 

Percentage of items 
Job relevance - Ttom** 



Percentage of college personnel choosing each 
reationse option aa approprinta for cumriciiliim 
Close Some Appreciable little 

Oaiallfil difffirftnCftfl diffi>rAn«»«.« aimilaritv 
0 6L5 30^ 7.7 



13 



Percentage of items 



Percentage of public school personnel 
who ratgd iittm etmtjmt. m relwant tn ^oh 
over over over over over 

smh €m> im, (m, 9CR(, 

97 92 85 74 46 



•Number of items =145 



2D 



18 



The results for the Educational Leadership test indicated that over 50% of the panelists 
viewed 95% of the questions as content appropriate. Based on the criterion of more than 70% of the 
panelists, the percentage of test questions judged content appropriate was 88%. A minimum of 
81% of the questions on each test was perceived as content appropriate by over 80% of the panel 
members. 

Relativft Rynphaseft of Tftst Topics and nurrirnlii The Content Review Panel members 
assigned to the test compared the mi^or content topics with the topics in the related college 
curriculum. Allowing a difference of 5% or less, the relative emphases were rated the "Same As," 
"More Than," or "Less Than." The responses were pooled by topic and weighted by percentage of 
test questions on the topic. A Difference in Relative Emphasis (DRE) Index was computed for the 
test The results are also presented in Table 4. The index range can be from a minimum of 0 
(very close similarity) to 100 (very little similarity). The DRE Index for the Educational 
Leadership test under study was 13.6. 

Overall Similaritv Rfttween Tpat.<i And nurrirtiln The Content Review Panel members 
made judgments about the similarity between the test as a whole and the related professional 
preparation programs. The choices for comparison were "Very Close Parallel," "Some 
Differences," "Appreciable Differences," and "Uttle Similarity." The percentages of panelists 
selecting each option are also depicted in Table 4 for each test reviewed. 

Considering close parallel and some difference as indicating relatively high congruence, 
the percentage for the test was 61.5%. Appreciable differences attained a 30.8% response choice. 

Job RftlfivflTicft PunAl RAciilf,ff 

The Educational Leadership test was reviewed by a Job Relevance Panel. The panel 
member reviewed each question and judged its relevance to the responsibilities of newly licensed 
school principals in Tennessee. Responses of "Critical," "Important, ' arid "Acceptable" were 
used to identify questions representing relevant knowledge and academic Bkilk. Responses of 
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"Questionable" and "Not Relevant" were defined as indicating irrelevance. The results of the job 
relevance ratings for the tests are also presented in Table 4. 

The ratings for the test revealed that more than 50% of the panel members) considered 97% 
of the questions on the test to be job relevant. Over 80% of the panelists perceived that 74% of the 
questions on the test were job relevant. 

Knowlftfigre Estinifltion Panel Results 

The members of the Knowledge Estimation Panel for the Educational Leadership test 
'.lade judgments independently on the percentage of minimally qualified applicants for licensure 
as principals in Tennessee who would know the answer to each question. Based on the response 
choi'^os (11%, 20%, 30%, 40%, 50%, 60%, 70%, 80%, and 90%) selected, the mean percentage for each 
qu6(:uon ^.as computed and adjusted upward to account for guessing the correct answer if 
unknown to the examinee. The sum of the adjusted percentages represented the estimated raw 
score mean for the test. The raw score mean was converted to a scaled score mean using 
conversion parameters provided. 

Table 5 contains a summary of the analysis on an estimated score for the test. Included 
are the raw score mean and the scaled score mean. Also included are total items and number of 
scored items for the test. 

Table 5 



SUMMARY OP THE ANALYSIS ON AN ESTIMATED SCORE FOR MINIMALLY QUALIFIED 
LICENSURE CANDIDATES FOR THE EDUCATIONAL LEADERSHIP TEST 





Items 




Mean 




T«tt 


Total/Scored 


Raw Score 


Scaled Score 


N 


Educational Leadership 


145/143 


67.837 


528.647 


33 



The scaled scores for each ETS subject-matter test can vary from a low of 250 to a high of 
990, a 740-point difference between the lowest and highest scores possible. A score cannot 
be compared directly with another subject-matter test for two reasons: first, each test is normed 
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independently on difTerent groups of examinees; second, the standard error of measurement, an 
index of the precision of test scores, varies to a considerable degree between tests. 

Concerning the test item analysis, the number of ETS choices, disagreements with ETS 
choices, additional competencies recommended by item, and maximum recom.nendations of a 
competency are shown in Appendix D. 

Standards Committee ArHnnr. 
The Standards Committee elected one of its members to serve as chair during the 
deliberations in order to take the formal actions required. I'he decisions on test validity, the 
recommendation of a minimum qualifying sccre, and matters related to the relationships between 
test items end competencies are reported below. 

Test VaV;ditv DeHfiinn 

The Standards Committee was presented all of the data collected and anal3^ed in 
performing the content review and job relevance review functions. (Knowledge estimation data 
were not released to the committee until the test validity decisions had been made.) The committee 
considered concurrently the content review results (test item, test topic, and total test levels) and 
job relevance review results for the Educational Leadership test independently in making 
decisions on test validity. Utilizing this approach, the committee concluded that the Educational 
Leadership test is valid to use in Tennessee for the licensure of principals. 

Rpcommender^ M iniTnuni Quftlilving Scorft 

The Standards Committee received the results of the knowledge estimation function for 
minimally qualified professional practitioners based on the judgments of the panel members who 
r'^viewed the Educational Leadership test . 

The data reported for the test with normative information were the following: items, total 
and scored; number of examinees; scaled score mean; standard deviation; standard error of 
measurement; and knowledge estimation scaled score mean. In addition, the values for scaled 
score means minus 1, 2, 3, and 4 standard errors of measurement were derived (Table 6). 
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Table 6 



NORMATIVE AND DERIVED DATA ON THE 
EDUCATIONAL LEADERSHIP TEST 



Term . Data 



Items 

TotaVScored 145^143 
Norms 

N 716 

Mean 619 

S.D. 102 

S.E.M. 29 

Knowledge Estimation 

Mean 529 

Mean - 

1S.E.M. 500 

2 S.E.M. 471 

3 S.E.M. 442 
4S.E.M. 413 



After a thorough review of the examinee performance data, the committee recommended 
that the Tennessee State Board of Education adopt a minimum qualifying score of 500 for the 
Educational Leadership test The committee further recommended that the score be reviewed in 
two years in order that Tennessee dat be considered. 

Actions Conceminp the Relationships of Test Itfims to Competencies 

After examining the data presented them, the committee concluded that there 
appeared to be some relationships between the items on the Educational Leadership test and the 
competencies deemed ^^ecessary for principal licensure in Tennessee. There were concerns 
expressed about items that had few references to competencies. 

Conclusion 

The actions of the Tennessee State Board of Education relating to the recommendations 
made by the Standards Committee are of interest and significance for personnel from teacher 
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preparation institutions and principals and others who will take the Educational Leadershi-^ test. 
Its actions represent an opportunity to set licensure score requirements for the test considered in 
this project in order to reflect curricular offerings of institutions involved in preparing 
professionals for administrative responsibilities. Establishing a required score reflects higher 
levels of accomplishment by Tennessee teacher education graduates and a stronger relationship 
with national norms, but necessitates avoiding an adverse effect on minority candidates. 

A concern of professional educators is to conduct periodic reviews of all tests required for 
licensure in T mnessee to ensure that the most recently-created and revised tests are analysed 
from the viewpoints of job relevance in K-12 schools and content correlation with college programs 
of study. Inasmuch as the contents of the tests are continually modified, changes in test items are 
made periodically, resulting in new versions of the tests. An on-going program of review will 
produce new sets of data on which to base recommendations of minimal scores for each new or 
revised instrument. Any consideration of a change in a minimum score will necessitate an 
analysis of the test for validity purposes, job relevance considerations, and knowledge estimation 
reasons. 
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APPENDIX A 
TEST CONTENT DESCRIPTION 



A-i 4G 



Approx • 
# of 
items 



Appro 
% of 
test 



EDUCATIOHAL LEADERSHIP : 

ADMINISTOATIOH AND SUPERVISION (EAS/41) 



I, Examinee demonstrates knowledge of leadership 

in the following areas of educational instruction: 

A. Determining needs: Pupil needs, conrounity 
expectations, national priorities, with 
knowledge of current research findings 

Curriculum design and instructional improvement: 
Determination of goals and objectives, instruc- 
tional methods, techniques and resources, 
strategies for implementing cxxrriculum decisions, 
and knowledge of relevant research data 

C. Development of staff and program development: 

Assessment of staff abilities, staff development. 
Evaluation: Strategies for change, types, 
methods, and procedures 



II • Examinee demonstrates knowledge of leadership in the 
following areas of educational administration: 

Organizational and operational features, business, 
fiscal and legal matters, governance and control 
features of school man^Q^sient 

III. Examinee indicates various individual and group 
leadership skills: 

Understanding and effecting change in individuals and 
groups through effective coonunication skills; utiliza- 
tion of coBBunity resources; creating and maintaining 
a positive and affective environment 



54 37% 



48 33% 



43 30% 



145 
items 
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APPENDIX B 

INSTRUCTIONS AND OVERVIEW OF TASKS TO BE 
PERFORMED BY MEMBERS OF THE 
CONTENT REVIEW PANEL 
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XNSTKUCTZONS FOR CONPLtTZMa TB COMTtNT XIVZIV POMf 



o Ua« only th« no. 3 (•oft«l«ftd) p«neil 9iv«a tp you. 
o MaJc« MCh aark 4«rk and eeapltttttXy fill th« eirela. 
o Oe net axtand aarkr eutsidc th« eirelss. 
e us« « eloan soft •n««r. Itum eoBpl«t«ly any changes you 
visli to 

o NaJM no stray aarks en the fezm. 

TIM itaaa In thia aaetien appaar at thm left of sida Ona. 
o ITS oaa OBly. Usa only as diraetad by tha panel leader. 



o faaal aMer. Pill in tha circles for the twe-difit nuaber 
assignad to your panel. 

o ZastmstioMl h&wml. Pill in on circle only next to the 
instrttctienal laval at mUdi yea taacH or provide other 



XD Minr pm in the circles far the thrae«-dif it nunber 
assignsd ts yoc as a panelist. 

MB. PiU la the sirds for aslc or fsHao. 

■MM. Plsass prist year nans in the space provided. 

Itftnicity. PiU in on cirds saly that best describee 



o Beta osUsstisn ants. PiU in tbs circles for the two»di«it 
nunbar for today's aantb* day and year. 

Verify that yea bnvc cntarod all rofoired infomatien. 



of ys« tasks is ta cnnine escb teet foostisn and jodfa 

jher or not a itadiiit in a teacher odoeatien profraa in 

Tennessee vsaM have had an epportanity ts ac^iiro the knovledfo 
and ae^4anic skUls to answer ths fosstion correctly. 



he Q IMO by IdMatlMftl Tattlat S«rriM. m tif^u n—rfd. 
MMrTMCiod UffviM U iS IfiiAl Opf«rtualcy lBfl«y«c. 



er|c best copv available 



XNSTROCTXONS FOR CONFtrrZMG THE COMTIMT SXVZIN POMC 



In MleiM your jud^BMt About Mch tMt question, conaid«r 
wh«th«r t&« knovl«d9« or «e«dmie sleills r«l«t«d to tho quostion 
would h«vo boon covorod in any of tbo coursoo nonolly tokon by 
•tudonto onrollod in on oppropriato training prograa. Xn sobo 
easos a eourso that is • proroquisito to ontoring a training 
prograa aay hava boon takon by ooao otudanto in col logo or in 
high school, whon such a couroo is ono in which tho knowlodgo or 
•kill rolatod to a particular quostion is taught, all studonts 
who would havo takon tho eourso, whothor in col logo or in high 
school, should bo considorod to havo had an opportunity to 
acquiro that knowlodgo or skill. 

Tho contont of a quostion should bo considorod to bo "taught" it, 
in your judgaont, at loast fO porcont of tho studonts would havo 
had an opportunity to acquiro that kiiowlodgo or skill through 
class loeturos or discussions, through laboratory assignaonts, 
through toxtbooks or thoir hoaowork i!issignaonts, or through othor 
outsido roading. You aro not to judgo irtiothor tho studonts 
would, in fact, havo loamod tho answor? XflM art gniy tQ Ittdgi 
whafchag tH* would Hav had an atmortunitv to aeouiro tho 

Thoro aro four rosponso eatogorios: 

Y * YOO 

P • Probably Yos 
Q « Probably Ho 
M • MO 

AS you oxaaino oaeh tost quostion and its ansvor, judgo whothor 
tho studonts in tho group with wliieh you aro eencomod would 
dofinitoly havo had an opportunity to ae^iro tba knowlodgo or 
acadoaie skills to anawor corroctly. Zf you think thoy would 
havo, fill in cirelo (Yoo) on tlM Contont loviow Fora^with a 
hoavy, dark aark so yoa cannot soo tho lottor in tho bubblo. If 
you think that tho otodonta would Bgohabiv havo had an 
opportunity to aeqoiTO tho knovlodgo of acadoaie skills, fill in 
tho cirelo "V* (Probably Yoa). Siailarly, fUl in tho ovals "Q" 
(Probably No) and m« («o) if yoo think atudonta would probrtlY 
not or dafttiif I'y not havo had an opportunity to aequiro tho 
knowlodgo or acadoaie akilla to anawor corroctly. 

Boforo you fill in a eirelo, ploaso aako euro that tho nuabor on 
tho form aatehoo tho nuabor of tho quostion in tho tost booklot. 
Zf you wish to change a rosponso, oraao cooplotoly your Urst 
choice so that your final judgaont will bo tho only ono piekod up 
by tho scanning aaehino. 

Xn aaking your judgaonts you are not to be eencomod about how 
aany questions you are assigning to each category; your 
responsibility is to apply your best judgaont in evaluating each 
question individually. 



er|c BESTCCPYAViSMlE « 



IHSTRUCTIONS FOR COMPLITIMC TKI COMTEMT MVIW FORM 

turn Mml wmmtinq, you will b« r«c«iving « Pin«U«t'« coM«nt 
£oJ «• yoS rSvi«*i fZ qu««tion«. Panelists should 
S?I on tho CoISoSt Shoot tho itoJ nu«bor of any toot quootion 



2h5t thorboliovo roquiroo rovi.ion or ro«ov«l J^o. tho toot. 
ThSSoh 111 toot quootiono hovo undorqono •xtonolv. oditoriol and 
••SSitivity rovlow, ponolioto should noto ony toot quootiono that 
appSar aabiguouo, awkwardly phraood, incorroct in ooao way, 
insonsitivs to a particular group of eandidatoo, or biasod in a 
way that would put a particular group of candidatoo at a 
disadvantago. Panolioto should provido an oxplanation of tho 
probloa for any toot quootion thoy idontify. 
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CONTENT REVIEW/KNOWLEDGE ESTIMATION PANEL 
OVERVIEW OF TASKS TO BE PERFORMED BY PANEL MEMBERS 



The study in which you have been asked to participate Is being conducted for 
the Tennessee Department of Education. You have been selected to serve nnfhn 
Content Review/Knowledge Estimation Panel. As a membe? of this paJllTySC 
UliLS*^"^?^ to perform two tasks, the first In the morning ses5iSn and She 
second In the afternoon session. An overview of each task follows. 

Content Revi^hf Overview 

One purpose of the study is to evaluate test content In relation to teacher 
education programs in Tennessee and to iudoe whethpr InnVT ^'n^" u?5"?r. 
QPMrtunltY to .rnnir. th. knowledge nr 

under miftw . During the morning session you will be asked to : 

(1) examine each test question and judge whether a student in 
teacher education programs In Tennessee would have had an 
opportunity to acquire the knowledge and academic skills to 
answer the question correctly. ^-^m* m 

(2) examine the description of test content used In develoolna 

K«SiI31^°"h°^ ^5* ^"^3t the extent to which the 

knowledge and academic skills associated with each content 
category are represented as an appropriate proportion of the 
test; I.e., whether each currlculuM content area Is appropriately 
InS'"!!??^*?* "n<*f«"P»»«l2ed. or Is overemphasized in the test. 
in\t^] !J*!.*^ J° ^^f^^^y important areas not represented 
In the test that could be reliably masured and should be Included. 

tJVlt^ 2f i;«P»*«"ntat1ve faculty menbers fro« throughout the state will 

S!dr?S?i?iui?i:*i!H^5*i*' judgments! howeJJ;. w?ll t. 

made Individually and Independently; nenbers of the same panel will not 

«;!:I*''»fiK! 5''°"?: ^"•y ^ the judg-tnts mide bj 

!i5fL??*?f*\ J"<'9«*nts of all Members of a pawrwlll be combined 

iill JrM.*? • *r"''y ^"^3^"^ *»»«"t each test queSliras 

well as the test as a whole. Summary results will be published In a f nal 
report describing the study and Its findings. 

IrJc^J/**! jnclosures In this mailing are Intended to help you prepare for 
this first task. If. in studying the materials, you find that you have 
questions about this task, be sure that they ari answered durlnJ the 
orientation session at the meeting. These enclosures are: 

Content RtYltw Fori (sample). This form Is to be used to record 
your Judgments about test questions. Please study the 
accompanying Instructions for completing the form and examine the 
sample form before you go to the panel meeting site. A Connents 
Form win be given to you at the panel meeting so you may write 
down comments about any aspect of the test. 

Copyright c 1990 by Educational Testing Service. All rights reserved. 
Educational Testing Service 1$ an Equal Opportunity Employer. 
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(2) Test Content Description . This enclosure identifies the major 
groups of topics covered by the test and indicates the relative 
emphasis given to each topic. At the meeting, you will be asked 
to judge whether each content area is represented as an 
appropriate proportion of the test, is underemphasized, or is 
overemphasized, and to record your judgment on the Test Content 
Description Review Form (described below). You will also be asked 
to identify important areas not represented in the test that could 
be reliably measured and should be included. 

(3) Test Content Descriotion Review Form (sample). You will use this 
form to your judgments regarding the content of the test. You 
will be asked to judge whether each content area listed in the 
Test Content Description is given about the same, more, or less 
emphasis in your curriculum compared to the emphasis given that 
area on the test. You will also be asked if there are major 
content areas not listed In the Test Content Description that 
should be added to the test. 



Knowledge Estimation Overview 

The second purpose of the study Is to estimate the test performance of 
minimally knowledgeable candidates for certification as beainnlna teachers or 
practitioners In the public schools of Tennessee. Your task will be to make 
JudgMnts about the difficulty of Individual test questions for persons who 
have the minimum level of knowledge necessary for competent performance as a 
beginning teacher or practitioner in Tennessee. Your judgments will be 
combined with judgiiients made by other panel members to derive an estimate of 
the probable test performance of this group of persons. 

School and college personnel from throughout the state will be assembled to 
make their judgments. Judgments regarding the difficulty of each test 
question will be made individually and Independently. Once each panel member 
has rated each of the items in the test, the Individual judgments of all 
members of the panel will be combined statistically to arrive at a summary 
judgment for a recommended passing score on the test. 

Two enclosures in this mailing are Intended to help you prepare for the 
Knowledge Estimation task: 

(1) Knowledge Estimation Form , This is a sample of the form that will be 
used to record your judgments about the test questions. Before going to 
the meeting site, please study the accompanying Instructions for 
completing the form, examine the enclosed sample form, and identify any 
questions about the task you would like to have answered during the 
orientation session. 
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(2) Examinee Knowledge About niustrat ive NTE Test Questions . This 

enclosure will be used at the panel meeting as part of an exercise to 
provide you with some experience in making judgments about the knowledge 
that characterizes defined groups of examinees for items which differ In 
difficulty. 

You have been asked to participate in this study because you are familiar with 
the knowledge and academic skills needed for competent performance as a 
beginning teacher In Tennessee. Before attending the panel meeting, however, 
you may want to draw upon local sources of information and to talk with 
colleagues. 



WHEN YOU COME TO THE MEETINfi. PLEASE BRIMS THIS Pir.rrf qf MATERIALS WITH YOl! 
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SoM tlB« wlXl b« pro'/ld^d «t tha data eoIl«etiott CMtar for you to eoaploto this term whon ers 
p^rionnol will b« <vailablo to aaiw«r quoationa^ ao plaaaa do noz racord any jud^Mitta on thia 
font baCorahand. 
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* tntoK tSio porsanta^aa liatad oa ^a Taat Coa^aat Oaaeriptiott form. 

Tha ebart abova ia a standard fosK that ia «aad to datasaisa to what da^raa tha talativa aaphaaia 
9ivaa aach to^ie ia tho MTB frofraaa Taata corraapoa^a ta tba ralativa aaphaaia of that topic 
i^ithia tha broad eurriotlw oiatliM of tha toaehor adaeatim aaquaoca at your i&atitution. 
Tha Roaaa nuaarala froa X to XX atovo oorraapood to tha Taat Coataat Topica Uatad ia tha Taat 
Coataat S^aeriptioa rora. Xofora ro« hofia to auto tha ooapariaoaa, plaaac racord tha 
pareaotafa fifura for aaeh topia aa Iiata4 oa th« rifht hasi aida of tha Taat Coataat Oaacription 
Fon. Xi you racord aaeh parcaotafa fifura« ha aura that tha suahar of tha C&iatant Topic on tha 
two foraa ia thQ aaaa. Bacaoaa ths aaaa fora io balof aaad for all taata« thak*a aay ha aora 
Contaat Topic auahan liatod oa thia foxa thaa thara ara topica ia tho taat which you ara 
raviawiaf « Flaaaa deuhla ehaok to aoo that tha laat auabar you aaaifa a parcaatafa to on thia 
foxa ia tha laat auahar oa tha fan. 

Ottca you hava fillo4 ia tha ooluna lahola^ *Fareaat of Taat''^ you oaa bo9ia avaluatiiif tha 
aaphaaia placod aa tho topiaa liatad ahov« ta tho ralativo aaphaaia placad oa taaaa tof^ica within 
tha broad eurrioulua of tho taaahar adueatioa aoquonca at your inatitutioa. If you daaira, you 
aay coaault wlfh oolloafuaa rafardinf thaaa quaatioaa hafara you anawar« hut tha raapoaaaa placad 
oci tisia foxa should rapraaant your boat judfaant. 

For aaeh topic ahoira« eircla tha latter nest to tho topic that indicatae your judfaant^ aa 

foliowat 

o Circle I if the topic ie fivea about the SAMS aaphaaia ia the teacher education curriculua at 
your institution aa ia the test. Oiere^sx^ saall parcontaf* diffarancaa« i*e.# diCfaracieea of 
S» or laaa. 

o Circle !i ifi the topic ia fivaa MOM aaphaaia ia ycur inatitution*a taecher education eurric* 
ulua thah in the taat. Cirale N eaiy if in your opieian the differenca ie freiater than S% 

O^. o Circle & if tha topic is fitron Lttf eaphaaia in your Uatitution'a teacher education rurric- 

ulua than in tha taet. Circle % only if In your opinion ^e, difference ia fraeter than 5». 

B4 



f AM TWO 

(1) li&««d on tho infoCTUtion you Hato ro^Ardin^ tho gonorAl contont topics covorod in this t«tt 
••loot tho option b*low that aott cloioly charActorisot your judgaont r^^Arding tho tiailarity 
b«tw««n this MTI tost And tho toachor oducation toquonco At your institution. Xndiccto your 
«ntw«r by plACin? «n X noxt to tho rotponao that you hava ehoaan, than uaa tha apaca balow to add 
any additional eo«Mnta you aay hr^a* 

^ — taat eontant topica paraxial tha taachar aducation aaquancaa at our inaitution 

vary cloaaly. 

■ Thara ara aoM diffaraneaa batvaan tha taat eontant topica and tha taachar education 

aaquancaa at our inatitutioa, but thaaa diffaraneaa do not appaar to ba appraciabla. 

^ Thara appaar to b« aoaa appraciabla diffaraneaa batwaan tha taat eontant topics and 

tha taachar aducation aaquaaea at our iaatitutioa. 

° Thara ia littla ai«ilarity b«twaao tha taat eontant topica and tha taachar aducation 

aaquaaea at our iaatitutioa. 

(2) Oaa tha apaca balov xiat aay Mjor coataat araaa ia your iaatitutioa'a taachar aducation 
aaquaac« that do act appaar to bm covarad ia tho Taat Coataat Oaacriptioa. 91aaaa nota that tha 
topica you vrita in ahould bo of approsiaataly tho laval of apocificity aa thoaa listad. Oo not 
liat topica that My ba aubauMd uadar oaa of tba cataqoriaa liatod — if you ara uncartain aa to 
whathar you ahould liat a topic hora# «#ait until you hava auainod tho packat of taat quaationa 
(you will raeaivo thaaa at tho data collactioa caatar) bafora doiaq eo. 




(3) aaa tha apaca bolov to iadieata aay odditioaal coaMaata you aay hava. 



faa«liat*a Siqnatura 
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INSTRUCTIONS FOR COMPLETING THE JOB RELEVANCE FORH 



Griddino Instructions 

0 Use only the No. 2 (soft-lead) pencil given to you. 

0 Make each mark dark and completely fill the circle. 

0 Oo not extend marks outside the circles. 

0 Use a dean soft eraser. Erase completely any changes you 

wish to make. 
0 Make no stray marks on the form. 



Panelist Information Section 

The items in this section appear at the left of Side One. 
0 ETS Ust Only. Do not fill in. 

0 Panel Kumter. Fill in the circles for the two-digit number 
assigned to your panel. 

0 Instructional Level. Oo not fill in. 

0 ID NuBibtr. Fill in the circles for the three-digit number 
assigned to you as a panelist. 

0 Stx. Fill In the circle for Halt or Feult. 

0 Num. Pltast PRINT your name In the space provided. 

0 Ethnicity. Fill In ONE circle only that best describes 
your background. 

0 Data Collection Date. Fill In the circles for the two-digit 
number for today's month, day and year. 

Verify that you have entered all required Information. 



Question Review Section 

Your task Is to «ake judg^nts about the extent to which the knowledge and 
academic skills tested by individual questions In the test under review are 
relevant to competent oerfomance as a bftoinnino teacher or other specified 
educationa l personnel In the schools of i'tnntsstt. 
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As you examine each test question and its answer, .iudae the extent to which 
the knowledge or aca demic skill tested in the question is relevant to 
competent performance of a beo innina teacher or other specified educational 
personnel in the schools in Tennessee. There are five response categories: 

C - Crucial 
I - Important 
A - Acceptable 
Q - Questionable 
N - Not Relevant 

When you have made your judgment, locate the column on the Job Relevance Form 
with the appropriate heading and fill in the corresponding circle (C, I, A, 
Q, N) with a heavy, dark mark so you cannot see the letter in the bubble. 
Before you mark a space, please make sure that the number on the form matches 
the number of the question in the test booklet. If you wish to change a 
response, erase completely your first choice so that you final judgment will 
be the only one picked up by the scanning machine. 

In making your judgments you are not to be concerned about how many questions 
you are assigning to each category; your responsibility is to apply your best 
judgment in evaluating each question individually. 

After you have finished making your judgments about the questions on a page, 
and again when you have finished the entire test booklet, please look over the 
questions and your responses to be sure that you are satisfied with your 
judgments. Also, check that the number of the last question for which you 
have recorded a judgment on the form corresponds to the number of the last 
question in the test booklet. Because the same form is being used for all 
tests, there may be more questions listed on the form that there are in the 
test booklet with which you are working. 

At the panel meeting, you will be receiving a Panelist's Comment Sheet for use 
as you review test questions. You should note on the Comment Sheet the item 
number of any test question that you believe requires revision or removal from 
the test. Though all test questions have undergone extensive editorial and 
sensitivity review, note any test questions that appear ambiguous, awkwardly 
phrased, incorrect in some way, insensitive to a particular group of 
candidates, or biased in a way that would put a particular group of candidates 
at a disadvantage. Please provide an explanation of the problem for any test 
question you identify. 
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JOB RELEVANCE/KNOWLEDGE ESTIMATION PANEL * 
OVERVIEW OF TASKS TO BE PERFORMED BY PANEL MEMBERS | 



The study In which you have been asked to participate is being conducted for the Tennessl 
Department of Education. You have been selected to serve on the Job Relevance/Knowledge 
Estimation Panel. As a member of this panel, you wi11 perform two tasks, the first in tt 
morning session and the second in the afternoon session. An overview of each task 
follows. 



a 



Job Relevance Overview 

The first purpose of the study is to review and evaluate test content in relation to the 
knowledge and academic skills which are relevant to teaching or practicing in the public^ 
schools of Tennessee. The corresponding task of your panel, to be carried out in the | 
morning session, will be to review each test question and to iudoe the extent to which tIS 
knowledge or acaiemic skills needed to answ er the question correctly are relevant 
competent perfom.^nce as a beoinnina teache r or practitioner in Tennessee in the specialU 
area covered by the test. % 

School personnel from throughout the state will be assembled to make their Judgments. TA 
judgments, however, will be made individually and independently; members of the same panfl 
will not confer as a group, nor will they be informed of the judgments made by other 
members. The judgments of all meiri)ers of a panel will be combined statistically to arrim 
at a suflmary judgment for the par.el about each test question, as well as for the test asB 
whole. The summary results will be published in a final report describing the study and* 
Its findings. 

Two of the enclosures in this mailing arc intended to help you prepare for the task I 
pertaining to Job Relevance. If, in studying these materials, you find that you have 
questions about the task, be sure they are answered during the orientation session at thtfi 
meeting. 1l 

(1) Test Content Description. This enclosure is provided to help you become mor^ 
familiir with the general content of the test you are to review. The 
enclosure identifies the major groups of topics covered by the test and ** 
indicates the relative enphisis given to etch. You will not be asked to make 
judgments about the topics and their rclitive emphasis. However, at the pan A 
meeting you will bt given a Comments Foni you may use to write down comments « 
about any aspect of the test. 

(2) Job Relevance Fom (sample). You will use this form and the accompanying B 
instructions to record your judgments about the test questions. Please studv 
the instructions for compltting the form and examine the enclosed sample for.,, 
before you go to the panel meeting site. ■ 

I 
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Knowledge Estimation Overview 

The second purpose of the study is to estimate the test performance of minimally 
knowledgeable candidates for certificatio n as beginning teachers or practitioners in th<* 
public schools of Tennessee. Your task will be to make judgments about the difficulty of 
individual test questions for persons who have the minimum level of knowledge necessary 
for competent performance as a beginning teacher or practitioner in Tennessee. Your 
judgments will be combined with judgments made by other panel members to derive an 
estimate of the probable test performance of this group of persons. 

School and college personnel from throughout the state will be assembled to make their 
judgments. Judgments regarding the difficulty of each test question will be made 
individually and independently. Once each panel member has rated each of the items in the 
test, the Individual judgments of all members of the panel will be combined statistically 
to arrive at a summary judgment for a recommended passing score on the test. 

Two enclosures in this mailing are intended to help you prepare for the Knowledge 
Estimation task: 

(1) Knowledge Estimation Form. This is a sample of the form that will be used to record 
your judgments about the test questions. Before going to the meeting site, please 
study the accompanying instructions for completing the form, examine the enclosed 
sample form, and identify any questions about the task you would like to have 
answered during the orientation session. 

(2) Examinee Knowledge About Illustrati ve NTE Test Questions . This enclosure will be 
used at the panel meeting as part of an exercise to provide you with some experience 
In making judgments about the knowledge that characterizes defined groups of 
examinees for items which differ in difficulty. 

You hav« b«en asked to parti cipatt In this study btcaust you art fail liar with the 
knowlcdgt and acadenic skills nttdcd for conpcttnt p«rformanct as a beginning teacher In 
Tena«s$«t. B«fortt mtnding tht panti Meting, howtvtr. you uy want to draw upon local 
sourcos of Inforaatlon and to talk with colleagues. 



Copyright c 1890 by Educational Testing Service. All rights reserved. Educational 
Testing Service is an Equal Opportunity Employer. 



erJc 



/ 



APPENDIX D 

INSTRUCTIONS AND OVERVIEW OF TASKS TO BE 
PERFORMED BY MEMBERS OF THE KNOWLEDGE 
ESTIMATION PANEL 



ERIC 



6.j 

D-l 



INSTRUCTIONS FOR COMPLETING THE KNOWLEDGE ESTIMATION FORM 



Qri^tf tnq Inttmetlorn 
o Use only the No. 2 (soft-lead) penci given to you. 
o Make each mark dark and completely ffl! the circle, 
o Do not extend marks outside the circles. 

o Use a dean soft eraser. Erase completely any changes you wish to make, 
o Make no stray marks on the form. 

Panellet Info rmation Section 

The Items In this sectkxi appear at the left cf Skle One. 

o ETS Use Only. Donotftiiin. 

o Test Number. Fill in the circles for the two-digit number to be assigned to your panel, 

o Instructional Level. Do not fll in. 

o lONumber. Fll In the dudee for the three<llglt number assigned to you as a panelist 

o Sex. FH In the cirde for Male or Female. 

o Name. Pleaae PRINT your name in the epace provided. 

o Ethnldty. Fl In ONE dide orty th« beet de«»toee your background. 

o Data ConecUoo Date. Fi in the drdes for the ti(»«HJIglt number for today's monlh. day and 
year. 

Please verify that you have entered alt required Momtatk)n. 
Qu—tion Revlaw Section 

Your task it to make judgmenlt about the dIBiculty of mdlvkjual test queetkxw for mInlmaMy 
knowledgeabiepefions in the field You w« be asked to dr«w upon your own experience to con«njci a 
hypottwticai group of persons, each of whom, in your judgment, has the iiMDUnU^^ 
unci «e«d«mir «kii« n«ee«Mn / for eomoetent oefformance as a beolnninQ educator in Tenneesee. 

As you read each tisi question and to answer, think of this group o«nMnalh^»^ 
tsM^hers or education personnel. '"^ n«rftK«o« of the orouo woiJd be able to identlfv Of to 
nfTh/iirttheangwr totheotieatlon . In making yoor estimate, assume that the examinees wolid not 
^SSbiSdlyifS^ You shoukl estimate the percentage who woiid know the 

answer without considering the possibllty that some additkxMl people migM pk:k 

lucky guess. 



ERIC 



Copyright e 1980 by »«c«tionil Ttitino Soviet. M right* r«««fvtd. EdocatiOMi Tt«ting Strviot is w Equal Oppoftuntty 
Employic. 

D-2 ^ • 



INSTRUCTIONS FOR COMPLETING THE KNOWLEDGE ESTIMATION FORM 



Page 2 



When you have made your estimate, locate the column on the Knowledge Estimation Fomi with the 
percentage heading (10. 20. 30, 40. 50. 60. 70. 80 or 90) that Is closest to your estimate and fit in the 
corresponding cirde on the form with a heavy, dark mark so that you cannot see the letter. Before you 
fili in a cirde. please make sure that the numt)er on the fom matches the number of the questkxi in the 
test txx>klet If you wish to change a response, erase your first choice completely so that your final 
judgmertt wll t)e the only one picked up by the scanning machine. 

Evaluate each test questkMi IndMdually. Do not t)e concerned atx)ut how many questk)ns you are 
assigning to the various percentage categories. 

After you have finished making your estimates about the questions on a page, and again when you have 
finished the entire test booklet pleasd look over the questions and your responses to be sure that you 
are satisfied with your estimates about the difficulty of the test questions. Also, check that the number of 
the last questkxi for which you have recorded an estimate on the form corresponds to the number of the 
last questk>n in the questton booklet Because the same form is being used for all tests, there may be 
more questk>ns listed on the form than there are in the test booklet with which you are working. 
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OVERVIEW OF TASKS TO BE PERFORMED BY 
KNOWLEDGE ESTIMATION PANEL MEMBERS 



Th« study in which you have been asked to participate is being conducted for the Tennessee 
Department of Education. You have tmn selected to serve on the Knowledge Estimation Panel. An 
overview of your taslc follows. 



Kncwtedqe EatSmatlon Overview 

The purpose of this study is to estimate the test oe fformance of mlnimaHv knowledQeable carxiidates for 
caftWcation a s beginning teachers or oractftioners in the public schools of Tennessee. Your tasl( wiU be 
to malce judgments about the difficulty of individual test questions for such candidates. A 'minimally 
l(nowledgeable* candidate is one who has the minimum ievei of Imowledge necessary for competent 
performance as a beginning teacher or practitioner in Tennessee. Your judgments will be combined with 
judgments made by other pane! members to derive an estimate of the probable test performance of this 
group of persons. 

School and coiiege personnel from throughout the state wll be assembled to malce their judgments. 
Judgments reotrding the ditKcutty of each test question wn be made individualty and independently. 
Once each panel member has rated each of tht items in the test, the individual judgments of aH 
members of the panel wl be combined statisricaMy to arrive at a summary judgment for a recommended 
passing score on the test 

Two endoeuree in this mating are intended to help you prepare for the Knowledge Estimation taste 

(1) Knowtedoe Estimation Form , This is a sample of the form that wl be used to recced your 
judgments about the test questions. Before going to the meeting sfte. please study the 
accompanying instructions for completing the form, examine the enclosed sample form, and 
idemtfy any questions about the task you would like to have answered during the orientatk)n 
session* 

(2) Examinee Knawtadoe About Wustmth/e NTE Teat Quaaaona , This endoeure wM be used at the 
panel meeting as part of an exerdee to provUe you with some experience in making judgmenu 
about the knowledge thot chanctertzes deAned groups of examinees fa Items which dHf er in 
diffictity. 

You have been asked to participate In this study because you are famtnar with the knowledge and 
aeademic akWa needed for comipeteM performance as a beginning teacher In Tennessee. Before 
attending the panel meetkig, however* you may want to draw upon local aoureea of informatkm 
and to talk wMi col e ag uee. 
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EXAMINEE KNOWLEDGE ABOUT ILLUSTRATIVE NTE TEST QUESTIONS 



PRACTICE EXERCISE 
(for UM at Pand Maating) 



As part of tha KnoMadga Estimation task, you wl! ba askad to draw upon your axparianca to construct a 
ISSlSSf^JS.'^^,*^'''?!!^ m you/judgmant. has ttia mlnlmnm i« val« m knnuH^^ 

acaflemK? itkiis tot comoatant offofmanf^. To halp you prapara for ttiis task, you wM ba askad to 
pwtteipata in an axardaa In which you astknata tha parformanca of a natkxwl sampla of NTE axamkiaas 

?^'f?'*^Jl!^L?i!!?^*^j£^*'^ Tha quastkxwara drawn from tha 

Cora Battary. tha portton of tha NTE Programa taatt that has l)aan takan pravk)U8ly l)y most 
ragardlaaaofthafiaMainwhk:hthaypiannadtotaach. 

For aach quastksn In tha practk:a axarcisa. you wll ba askad to astlmata tha parcantaga of axaminaas in 
two catagoriaa who knaw tha answar. Thasa two catagoriaa of axaminaas ara daflnad as foUowt: 

(1) thoaawhoaascoras.whla not tha kjwast for tha total group, warabalowavaraga (between 
tha 20lh and 40th parcantlaa): and «*v«n 

parcantlaa). 

Youwlraco(dyourattimataaonthafbnnprintadonthabM:kofthiapaoa. After you have made aach 
artmrta. k»cata tha column on tha fbmi with tha pareantaga haadkig (10. 20. 30. 40. 50. 80. 70. 80 or 90) 
thit la doaaat to your aathnata and fl in tha corraapondkig space on tha (bna 

NaMt you wl be aaked to ectknate the percentage of nMnMly knowladgeabia candMatea lor taachkxi 
cartMcctee in the state who wouM know the answer. 



of thoee who knaw the 
the figuraa you see 



After ycy have completed your eatimatea. you wl be givwi the 'acturi* 
cocrect ancweri. (The true flgwaa have been adjusted to account fbr kjcfcy , 
wl be statistical eatimatea of tha pereartagea who r«My knew the anawars.) 

The axerdae Is not Intended to help you fonmiate your oonoepdon of the mkHmely knowtedgeabfe 
exammee; rather. H la deaigned to giva you aoma experience In maMng ]udgmenla about tha knowledge 
dwnonstrated by defined gnoupe of examinees fbr questkins which diner In dMIctAy. 



a«towAvwag«araiip 
(2(Mi • 40tH PwcwitM) 



Abow* awng* Oreup 
(aoM'aodiPMMUto) 




PcrecntUa 



Strewn — wntam a 
M o n n n Dtatriauten 
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PRACTICE EXERCISE 
ANSWER SHEET 



Performance of Below Average 
Test Takers (20th to 40th Percentile) 



Your Estimates 

1. ®®®®®©©©© 

2. ®®®®®®®®® 

3. ®®®®®®®®® 



Actual Data 



1. 
2. 

3. 





Performance of Above Average 


Test Takers (60th to 80th Percentile) 


Your Estimates 


Actual Data 


L ®®®®®®®®e 


1. 


2. ®®®®®®®®® 


2. 


3. ®®®®®®®®® 


3. 



Performance of Minimally Knowledgable 
Test Takers in This State 



Your Estimates 

1. ®®®®®®®®® 

2. ®®®®®©®®® 

3. ®®®®®®®®® 
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SAMPLE QUESTIONS FOR 
KNOWLEDGE ESTIMATION EXERCISE 



Teachers who move to different school systems within a state will M that which 
of the following are most likely to be different? 

(A) Ages during which children musX attend school 

(B) Requirements for oertificstion of teachers 

(C) Laws regarding ootlective tsargaining 
(0) Provisions for teacher tenure 

* Grading practices 



Federal court interpretations of the Constitution suggest that a school can subject 
& studem to long^erm suspension or expulsion only if the student 

(A) has participated biwia^Mty that is legafiy a crime 

(B) it proven to be cognizant of regulations forbklding Ns or her act 

(C) has pailicipatad In an activtty that is toqxessly forbidden by school poOcy 
that it avalable in written form to both students and parents 

(D) has been given a hearing and it Judged guMy of the charges by a panel 
consisting of a school adminiitralor, taadilsrt. and other students 

*(E) is inlbmnedc.^ the charges, given the right to a hearing, and told of the right 
to appeal 



(continued on back) 
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Iht ciftoon (above) It mskir^g « sittamant of ths pubSc's reaction to 

(A) tha dropping of tha atomic bomb on Hiroshima and Nagasaki 

*(B) thaaclivitiM of Senator Joseph KtoCartliy in 1ha19S0*s 

(C) President Kennedy's actions in the Cuban mis^ crisis 
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APPENDIX E 
MATERIALS FOR NOMINATING PANEL MEMBERS 



INFORMATION FOR PANEL MEMBER NOMINATION 
TENNESSEE TEST VALIDATION PROJECT 



for 

Institutions of Higher Education 
and 
School Districts 



Bureau of Educational Research Service 
College of Education 
Memphis State University 
Memphis, TN 38152 
(901)678-2362 
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I. The Test 

The Educational Leadership test has been selected for complete validation for 
possible use in Tennessee. 

A. The test is an NTE test. Several NTE Specialty Area Tests are currently 
used for initial teacher licensure in Tennessee. These tests have been 
produced by Educational Testing Service, Princeton, New Jersey. 

B. This Educational Leadership test measures understanding of the content 
and methods applicable to the particular specialized area. The test is 
applicable for persons who typically are completing programs w'th a 
specialization in the principalship. 

n. The Validation Study 

To carry out the validation study, public school and college personnel will be asked 
to serve on the Knowledge Estimation Panel, Job Relevance Panel, and Content 
Review Panel. 

A. The Knowledge Estimation Panel will provide estimates of the percentages 
of minimally-knowledgeable candidates who would be expected to know the 
answers to individual test questions. The information provided by the 
panel will be used to develop a statistical estimate of the score that a typical 
minimally-knowledgeable candidate might be expected to achieve on each 
of the tests undergoing validation. 

B. The Job Relevance Panel will make judgments about the relationship 
between test item content and responsibilities of first-year practitioners. 

C. The Content Review Panel will judge whether or not at least 90% of the 
graduates in a given field would have the opportunity to acquire the 
knowledge or skills to answer an item correctly. 

Orientation, support services, and materials will be provided by Educational 
Testing Service staff. Memphis State University personnel will be in charge of the 
enlistment of panelists. 

The analysis of all the data will culminate in a report to the Tennessee State 
Department of Education. Information gained fi-om the work of the panel will be 
used to establish minimum scores on the tests for initial licensure and 
endorsement. 

in. Criteria for Panel Member Eligibility 

Criteria used in selecting panel members will include the following: 

A. Currently serving in the public schools in the area for which they are being 
recommended m currently serving as a faculty member in a college or 
university offering one or more approved teacher education programs. 

B. A minimum of two years of experience as defined in A above. 

C. Tennessee licensure for public school nominees. 

IV. Composition of a Panel 

The panels will be composed of both public school and college personnel. 
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A. Number . Approximately equal numbers of public school principals and 
college faculty members will participate in the validation study. 

B. GeQ^aphi<^, Representation of School Personnel. The geographic 
distribution of the public school personnel will provide roughly proportional 
representation across the eastern, middle, and western parts of the state. 

C. Representation of Higher Edueation Institutions. The distribution of 
personnel from teacher education institutions will represent the various 
types of institutions and will take into accoimt the productivity of approved 
programs. 

D. Other Partors . In the selection of panel members from the nominees, 
attention will be given to racial/ethnic groups. 

V . Time Schedule for the Validation Study 

Our selection of the panel members must be completed as soon as possible. 
Notification of appointment, acceptance by the nominee, and distribution of 
preliminary materials will be accomplished as nominations arrive. Individuals 
nominated will attend a full-day meeting in either Knoxville or Nashville in July, 
1992 at 8:30 a.m. and lasting no later than 4:00 p.m. 

VI. Other Pertinent Information 

A. With few exceptions possibly due to special circumstances, the number of 
nominees from any local school district or higher education institution who 
will be selected to serve on the panels is expected to be almost all of the 
nominations submitted. 

B. Each panel member will be asked to serve one day. 

C. Financial support for both travel, lodging, and meals will be provided by the 
validation study contractor - Bureau of Educational Research Service, 
Memphis State University. State of Tennessee rules concerning travel will 
be applicable. 

D. Honoraria cannot be provided. However, if the employing school district of 
a public school staff member requires that funds be provided to pay for a 
substitute, appropriate arrangements can be made by the contractor's staff. 

Contractor contact person: 
John R Petry 

Bureau of Educational Research Service 

College of Education 

Memphis State University 

Memphis, TN 38152 

(901)67&*2362 

FAX90147fr4206 
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Bureau of Educational Research Service LEA FORM 

302 Ball Education Building, Memphis State University 
Memphis, TN 38152 



FORM TO SUBMIT A NOMINATION FOR APPOINTMENT TO PANELS 
FOR THE TENNESSEE TEST VALIDATION STUDY 

1. NAME: 

2. TELEPHONE NUMBERS: Home ( ) ; Business ( ) 

3. ADDRESSES: 

BUSINESS School/District 

Number, Street 

City, State, Zip 

HOME Number, Street 

City, Stat«, Zip 

4. RELEVANT EXPERIENCE (PRESENT POSITION FIRST): 
EMPLOYKH POSTTTON f state .tor tttt .va dates 



5. OTHER DEMOGRAPHIC DATA (ANSWER ALL THAT APPLY): 

A. M ALE ^FEMALE 

B. (Optional) _WHITE ^BLACK ^HISPANIC 

AMERTHAN INDIAN OTHER (Specify) 

C. AGE: 25 or less __2&^ _31-35 _36-40 

^41-45 __46^ _51-55 _56 or more 

D. EDUCATIONAL BACKGROUND: 

DEGREE YEAR iNSTiTimnN MAJOR 



E. Certification (check one or more): 

.^General Science (014) ^History (021) 

— ^Government (023) I_School Social Worker (107) 
Visually Impaired (108) 

NAME OF SCHOOL DISTRICT NOMINATING 

SIGNATURE/TITL E 

ADDRESS 

No^treet City State Zip 

^ E-5 



Bureau of Educational Research Service IHE FORM 

302 Ball Education Biiilding, Memphis State University 
Memphis, TN 38152 



FORM TO SUBMIT A NOMINATION FOR APPOINTMENT TO PANELS 
FOR THE TENNESSEE TEST VALIDATION STUDY 



1. NAME: , -. 

2. TELEPHONE NUMBERS: HomeLJ ; Business LJ. 

3. ADDRESSES: 

BUSINESS School/District 

Number, Street . 

City, State, Zip . 

HOME Number, Street 

City, State, Zip . 



4. RELEVANT EXPERIENCE (PRESENT POSITION FIRST): 

EMPLOYER POSmON f IF TEACHER. liATES 

STATE i 



5. OTHER DEMOGRAPHIC DATA (CHECK ALL THAT APPLY): 

A. ^MALE ^FEMALE 

B. (Optional) W HITE BLACK ^HISPANIC 

AMERICAN INDIAN ^OTHER (Specify) 



C. AGE: __25orle88 _26^ _31-35 _36-40 

_41-45 46-50 _51-55 56 or more 

D. EDUCATIONAL BACKGROUND: 

DEGREE YEAR INSTITUTION MAiIQfi 



NAME OF INSTITUTION NOMINATING 

SIGNATURE/TITLE 

ADDRES S 

NoVStreet City State Zip 



NOMINATIONS RESPONSIBILITY FORM 



It is most important that you let us know who will be responsible for the nomination of 
panelists. Please provide us the information requested on this form and return it with your 
nominations by return mail. 

Full Name 

Title/Position 

Business Telephone Number 

Full Name 

Title/Position 

Business Telephone Number ^ 

Full Name 

Title/Position 

Business Telephone Ntmiber 

Full Name 

Title/Position 

Business Telephone Number 

Submitted by: ^ 



Institution/School District 
E-7 



TENNESSEE 

STATE DEPARTMENT OF EDUCATION 
OFFICE OP COWMISStONE.'^ 
NASHViLLE, TENNESSEE 37243^75 

May 29. 1992 



Or. Nathan L. Essex, Dean 
College of Education 
Memphis State University 
215 Education Building 
Memphis, Tennessee 38152 

Dear- Or. Essex: 

In Inplewentlng tf^e State Board of Education's Policy for the Principal 
In Tennessee's Schools and respective Licensure Standards, we will be 
working with FJenphls State University and Educational Testing Services 
regarding validation the National Teacher's Ex»nination Specialty 
Area Test for Admiriistrators (NTE Educctional Leadership: 
Administration and Supervision Test). 

We must begin the validation process new to meet the Policy's Effective 
date of July 1, 1994^ which will re<5uir(E individuals einployed for the 
first time as a principal to neet the test requirements as specified. We 
have an opportunity for higher education to have input into the 
validation of the P-sMncipal's Test. 

I invite you to send a representative from your Department of 
Educational Administrfition to attend a fseting scheduled for either July 
16 or July 23. These mettinss a»*e being held fn conjunction with a 
previously scheduled Tennessee Acadewy for School Leaders session with 
principals from across the $UU for the purpose of validating this 
test. Choose only one meeting as they are repetitive in content and 
process. 

The July 16th meting will be held in K.^oxvllle, Room C of the Statt 
Office Building located behind the ifoirld's Fair HoTiday Inn on HenUy 
Street. Tht July 23rd nteting will be iteld in Nai^hvinn 1» the Cordell 
Hull Building, Room Cl-124. Both days are schtdylcd fvvm 8:M a.«. 
until the task is completed. 

If you wish to send a representative to this <«attt collection session for 
validation, please call tfendy Siebert at (615) 7<l->6058 as soon as 
possible. Ms. Siebert will be able to provide you other particulars 
regarding directions and lodging choices if needed. 



CO 



Dr. Nathan L. Essex 
Hay 29 » 1992 
Page 2 



Do not hesitate to call me at (615) 741-1441 if you have any questions 
regarding this process or policy. Thank you for your interest and 
participation as we move toward implementing the Administrator Policy. 

Si nee rely » 



Dr. Connie J. Smith 
Director 

Division of Teacher Education and Accreditation 
CJS:wrs 

cc: Dr. Tom Valesky, Chair 

Dr. Nebraska Mays» Deputy Commissioner 
Dr. Harry Bownan, Memphis State University 
Mr. J.T. Stewart, Educational Testing Services 
Ms. Deborah Gilliamt Director, Research and Development 
Dr. Susan Hudson, Director, Teacher Licensing and Career Ladder 
Certification 

Ms. Betty Long, Director of Planning and Special Programs 

Dr. Elaine Willers, Director, Tennessee Academy for School Leaders 
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APPENDIX F 



LIST OF PERSONS CONSENTING TO BE LISTED 
AS PANELISTS IN THE VALIDATION STUDY 



PANEL MEMBERS CONSENTING TO BE LISTED 
AS PARTICIPANTS IN THE VALIDATION STUDY 



1. 


Hugh D. Adams 


2. 


Fred Bedelle, Jr. 


3. 


David Benny 


4. 


Charles W. Burkett 


5. 


Rosella Carruth 


6. 


Eloise Dabney Cheatham 


7. 


Bernard Childress 


8. 


Linda CofTey 


9. 


Mary Jane Connelly 


10. 


Joe Dent 


11. 


Roy D. Dukes 


12. 


David F. Green 


13. 


Mary Anne Halt 


14. 


Danny N. Hanson 


15. 


David Heath 


16. 


Ray W. Hogan 


17. 


Sheryl Kerley 


18. 


Don Lambert 


19. 


Sam Lucas 


20. 


Bob McElrath 


21. 


Sam Miles 


22. 


Joe Moses 


23. 


Alfred L. Motlow, Sr. 


24. 


Joseph Murphy 


25. 


Carl Seale 


26. 


Vivian L. Sims 


28. 


Mary R. Walker 


29. 


Colbert W. Whitaker 


30. 


Vivian Woods 


31. 


Gary W. York 



Marshall County Schools 

Lincoln Memorial University 

Maryville City Schools 

Sast Tennessee State University 

Polk County Schools 

Maury County Schools 

Maury County Schools 

Scott Coiinty Schools 

University of Tennessee, Knoxville 

Knox County Schools 

Marshall County Schools 

Polk County Schools 

Chattanooga City Schools 

Marshall County Schools 

Weakley County Schools 

Metro Davidson County Schools 

Knox County Schools 

Austin Peay State University 

Memphis State University 

East Tennessee State University 

Trenton City Schools 

Trevecca College 

Memphis City Schools 

Vanderbilt University 

University of Tennessee, Martin 

Giles County Schools 

Hamilton County Schools 

University of Tennessee, Chattanooga 

Hamilton County Schools 

Williamson County Schools 
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APPENDIX G 



LETTER OF NOTIFICATION AND SELECTED MATERIALS 
SEOT TO AND/OR RECEIVED FROM PANEL MEMBERS 
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Julys, 1992 



Thank you for participating in the Test Validation study conducted by 
personnel from Memphis State University and Educational Testing Service, 
Atlanta for the Tenn essee State Department of Education. 

Attached is a sheet stating where your group will meet and giving 
particulars about how your expenses will be taken care of. You will receive 
from TSDE some information concerning the Content Review task and the 
Knowledge Estimation task that will prepare you for what to expect. Please 
bring the package with you. 

You will bIzo be sent another set of materials concerning a third task that 
you will asked to complete, which relates to the list of 15 knowledges, 
skills, and abilities deemed necessary for licensure standards for 
admini str ator s. 

If you have need of additional information, call me at 901-678-3407. 

Sincerely, 



John R. Petry 

Research Associate 

Bureau of Educational Research 

Service 

Memphis State University 

Enclosures: 

Panel Member Information Sheet 
Leadership Meeting Attendees 
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PANEL MEMBER REPLY FORM 
TENNESSEE VALIDITY STUDY OF THE EDUCATIONAL 

LEADERSHIP TEST 



Check the appropriate box to indicate whether or not you can serve as a 
panel member. Complete the form to provide the information requested. 

i I I» , will participate in the 

(Print your name) 

Tennessee Validity Study of the Educational Leadership Test 
Date of Meeting: City: 
I 1 I will not be able to participate in the study. 



Signature: 
Employer: 



Title or 

academic rank: 



Test assigned: 



(See Panel Member Information Form) 



Estimated round trip mileage from your location to the test validation site: 



Would you please complete this form and mail it in the enclosed postage- 
paid envelope AS SOON AS POSSIBLE. We need to receive your reply by 
retum mail. Thank you. 
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FORMB 



NTE DISCLOSURE POLICY AND RELEASE FORM 

I understand the importance of protecting the security of the NTE 
Educational Leadership test. I accept responsibility for the proper 
safeguarding of these confidential tests and agree to the following 
conditions: 



1. The copy of the test will remain in full view of the ETS 
representative during the entire inspection period. 

2. My copy of the test will be returned to the ETS agent each 
time I leave the meeting room, and at the end of the 
inspection period. 

3. Neith er I nor any member of my family will take an 
NTE test for a period of one year following the inspection 
without requesting permission from ETS in writing at 
least six weeks prior to a scheduled test date. 

4. I will not take notes, or otherwise record, copy, or 
disclose items or responses during or after the 
inspection. 

5. If for any reason the ETS representative must leave the 
meeting room, all test copies will be gathered and 
returned to the ETS agent. 



Signed 
Institution 
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APPENDIX H 

LIST OF MEMBERS OF THE STANDARDS COMMITTEE 
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MEMBERS OF THE STANDARDS COMMITTEE 
TEST VALIDATION STUDY, SEPTEMBER, 1992 

Joe Cornelius, Associate Professor, Educational Administration, Tennessee State University, 
Nashville 

Pat Gammon, Mid-Cumberland Region Director, Tennessee PTA, Hendersonville 

Tom George, Professor, Associate Dean, College of Education, The University of Tennessee, 
Knoxville 

Beverly Heame, French Teacher, Central-Merry High School, Jackson-Madison County Schools, 
Jackson 

Charles Jenkins, President-Elect, School Board Association, Pulaski 

B. J. Naylor, Vice-President for Academic Affairs, Freed-Hardeman University, Henderson 
Relzie Payton, Instructional Supervisor-Middle Schools, Shelby County Schools, Memphis 
Dan Russell, Directcr of Human Resources, Johnson City Schools, Johnson City 
Bettye Triplett, Consultant, Special Projects, Metro Public Schools, Nashville 
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LIST OF ATTENDEES 
STANDARDS COMMITTEE MEETING 
Nashville, Tennessee 
September 17, 1992 

I 

i 

Joe Cornelius, Associate Professor, Educational Administration, Tennessee State University, 
Nashville 

I Pat Gammon, Mid-Cumberland Region Director, Tennessee PTA, Hendersonville 

i Tom Greorge, Professor, Associate Dean, College of Education, The University of Tennessee, 

Knoxville 

Beverly Heame, French Teacher, Central-Merry High School, Jackson-Madison County Schools, 
^ Jackson 

1 

[ Charles Jenkins, President-Elect, School Board Association, Pulaski 

I 

^ B. J. Naylor, Vice-President for Academic Affairs, Freed-Hardeman University, Henderson 

I Relzie Payton, Instructional Supervisor-Middle Schools, Shelby Coxinty Schools, Memphis 

Dan Russell, Director of Human Resources, Johnson City Schools, Johnson City 
Bettye Triplett, Consultant, Special Projects, Metro Public Sch. -Is, Nashville 



Qbaerverg 

Deborah Gilliam, Tennessee Department of Education 

Betty Long, Tennessee Department of Education 

Susan Hudson, State Board of Education 

Nancy Simpkins, State Board of Education 

Elaine Willers, Tennessee Department of Education 

John R. Petty, MSU College of Education Staff 

Tom Snider-Lotz, Educational Testing Service Representative 
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